tokenizer - "pinyin" do g2p for only chinese characters, need .txt vocab_file - "char" for char-wise tokenizer, need .txt vocab_file - "byte" for utf-8 tokenizer - "custom" if you're directly passing in a path to the vocab.txt you want to use
(dataset_name, tokenizer: str = "pinyin",tokenizer_path="Emilia_ZH_EN_pinyin/vocab.txt")
source not stored for this graph (policy: none)
no outgoing calls
no test coverage detected