Instantiate a BertTokenizer from pre-trained vocabulary files.

Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args:

Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args:

Runs basic whitespace cleaning and splitting on a piece of text.

Runs basic whitespace cleaning and splitting on a piece of text.

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (vocabulary

Performs invalid character removal and whitespace cleanup on text.

Performs invalid character removal and whitespace cleanup on text.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Checks whether `chars` is a whitespace character.

Splits punctuation on a piece of text.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

Adds whitespace around any CJK character.

Loads a vocabulary file into a dictionary.

Loads a vocabulary file into a dictionary.

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (vocabulary

Constructs a BertTokenizer. Args: **vocab_file**: Path to a one-wordpiece-per-line vocabulary file **do_lower_case**:

Constructs a BasicTokenizer. Args: **do_lower_case**: Whether to lower case the input. **never_split**: (`optional`)

Constructs a BertTokenizer. Args: **vocab_file**: Path to a one-wordpiece-per-line vocabulary file **do_lower_case**:

Constructs a BasicTokenizer. Args: **do_lower_case**: Whether to lower case the input. **never_split**: (`optional`)

Converts an index (integer) in a token (string/unicode) using the vocab.

Converts an index (integer) in a token (string/unicode) using the vocab.

Converts a token (str/unicode) in an id using the vocab.

Converts a token (str/unicode) in an id using the vocab.

Checks whether CP is the codepoint of a CJK character.

Checks whether CP is the codepoint of a CJK character.

Checks whether CP is the codepoint of a CJK character.

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (string) in a single string.

tokens=[x1,x2,...]

text="...."

Instantiate a BertTokenizer from pre-trained vocabulary files.

Save the tokenizer vocabulary to a directory or file.

Save the tokenizer vocabulary to a directory or file.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi