total number of tokens

Support for backwards-compatible `past_key_value` length, e.g. `len(past_key_value)`. This value corresponds to the number of layers

Size of the full vocabulary with the added tokens.

Sample index mapping is a 2D array with sizes [number-of-samples + 1, 2] where [..., 0] contains the index into `doc_idx` and [..., 1] is the

Converts a token (str) in an id using the vocab.

Simple forward step with cross-entropy loss.

Initialize the weights.

Initialize the weights if they are not already initialized.

Prepared max and min length in generation configs to avoid clashes between similar attributes

This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`]

Returns a tokenized string.

Text generation arguments.

Text generation arguments.

Text generation arguments.

Add token-type embedding. This function is provided so we can add token-type embeddings in case the pretrained model does not have it.

Add token-type embedding. This function is provided so we can add token-type embeddings in case the pretrained model does not have it.

Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added

Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_toke

`List[str]`: A list of the unique special tokens (`'<unk>'`, `'<cls>'`, ..., etc.). Convert tokens of `tokenizers.AddedToken` type t

`List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`'<unk>'`, `'<cls>'`, etc.), the order has nothing to do with the

`tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend.

Broadcast data from rank zero of each model parallel group to the members of the same model parallel group. Arguments: keys: list of

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens