Decode text to chars or BPE Inplace operation Args: data: Iterable[{key, wav, txt, sample_rate}] Returns: Iterable[{key, wav, txt, tokens, label, sample_rate}]
(data, get_tokenizer, allowed_special, mode='train')
| 292 | yield sample |
| 293 | |
| 294 | def tokenize(data, get_tokenizer, allowed_special, mode='train'): |
| 295 | """ Decode text to chars or BPE |
| 296 | Inplace operation |
| 297 | |
| 298 | Args: |
| 299 | data: Iterable[{key, wav, txt, sample_rate}] |
| 300 | |
| 301 | Returns: |
| 302 | Iterable[{key, wav, txt, tokens, label, sample_rate}] |
| 303 | """ |
| 304 | tokenizer = get_tokenizer() |
| 305 | |
| 306 | for sample in data: |
| 307 | assert 'text' in sample |
| 308 | sample['text_token'] = tokenizer.encode(sample['text'], |
| 309 | allowed_special=allowed_special) |
| 310 | yield sample |
| 311 | |
| 312 | |
| 313 | def shuffle(data, shuffle_size=10000, mode='train'): |
no test coverage detected