Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the

Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the

Returns the vocabulary as a dict of {token: index} pairs. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` wh

Indicate if this BatchEncoding was generated from the result of a PreTrainedTokenizerFast Returns: True if generated from subclasses

Load tf checkpoints in a pytorch model.

Args: buf: the output of `dumps`.

Mask token (string). E.g. when training a model with masked-language modeling. Log an error if used while not having been set.

Id of the mask token in the vocabulary. E.g. when training a model with masked-language modeling. Log an error if used while not having been set.

Compute the bounding boxes around the provided masks The masks should be in format [N, H, W] where N is the number of masks, (H, W) are the spati

Kept here for backward compatibility. Now renamed to `model_max_length` to avoid ambiguity.

Get number of (optionally, trainable) parameters in the module.

Returns the number of added tokens when encoding a sequence with special tokens. Note: This encodes inputs and checks th

Padding token (string). Log an error if used while not having been set.

Id of the padding token in the vocabulary. Log an error if used while not having been set.

Id of the padding token type in the vocabulary.

Prune a Conv1D or nn.Linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_gr

Resize input token embeddings matrix of the model if new_num_tokens != config.vocab_size. Take care of tying weights embeddings afterwards if

Save checkpoint to file. The checkpoint will have 3 fields: ``meta``, ``state_dict`` and ``optimizer``. By default ``meta`` will contain vers

Save a model and its configuration file to a directory, so that it can be re-loaded using the `:func:`~transformers.PreTrainedModel.from_

Save the tokenizer vocabulary files together with: - added tokens, - special-tokens-to-class-attributes-mapping,

Save a configuration object to the directory `save_directory`, so that it can be re-loaded using the :func:`~transformers.PretrainedC

Save the tokenizer vocabulary to a directory. This method does *NOT* save added tokens and special token mappings. Please us

Separation token (string). E.g. separate context and query in an input sequence. Log an error if used while not having been set.

Id of the separation token in the vocabulary. E.g. separate context and query in an input sequence. Log an error if used while not having been set.

A dictionary mapping special token class attribute (cls_token, unk_token...) to their values ('<unk>', '<cls>'...) Convert to

A dictionary mapping special token class attribute (cls_token, unk_token...) to their values ('<unk>', '<cls>'...) Keep the t

Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a CharSpan Name

Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as:

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi

Unknown token (string). Log an error if used while not having been set.

Id of the unknown token in the vocabulary. Log an error if used while not having been set.

Size of the base vocabulary (without the added tokens)

Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are re

Get the encoded token span corresponding to a word in the sequence of the batch. Token spans are returned as a TokenSpan NamedTuple