Types & classes — github.com/Morizeyao/GPT2-Chinese

CodeHub MCPcopy Create free account

hub / github.com/Morizeyao/GPT2-Chinese / types & classes

Types & classes8 in github.com/Morizeyao/GPT2-Chinese

⨍ Functions 76 ◇ Types & classes 8

↓ 1 callersClassBasicTokenizer

Runs basic tokenization (punctuation splitting, lower casing, etc.).

tokenizations/tokenization_bert_word_level.py:206

↓ 1 callersClassBasicTokenizer

Runs basic tokenization (punctuation splitting, lower casing, etc.).

tokenizations/tokenization_bert.py:203

↓ 1 callersClassEncoder

tokenizations/bpe_tokenizer.py:31

↓ 1 callersClassEncoder_SP

tokenizations/bpe_tokenizer.py:94

↓ 1 callersClassWordpieceTokenizer

Runs WordPiece tokenization.

tokenizations/tokenization_bert_word_level.py:356

↓ 1 callersClassWordpieceTokenizer

Runs WordPiece tokenization.

tokenizations/tokenization_bert.py:339

ClassBertTokenizer

r""" Constructs a BertTokenizer. :class:`~pytorch_pretrained_bert.BertTokenizer` runs end-to-end tokenization: punctuation splitting + wordpie

tokenizations/tokenization_bert_word_level.py:89

ClassBertTokenizer

r""" Constructs a BertTokenizer. :class:`~pytorch_pretrained_bert.BertTokenizer` runs end-to-end tokenization: punctuation splitting + wordpie

tokenizations/tokenization_bert.py:86