Initialize the weights

Initialize the weights.

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTraine

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer}

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTraine

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer}

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrai

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer}

Tokenize a string.

Tokenize a string. Args: - add_prefix_space (boolean, default False): Begin the sentence with at leas

Tokenize a string.

Tokenize a string given language code. For Chinese, Japanese and Thai, we use a language specific tokenizerself. Otherwise, we use Moses.

Tokenize a string. return_unicode is used only for py2

Add a dictionary of special tokens (eos, pad, cls...) to the encoder and link them to class attributes. If special tokens are NOT in

All the additional special tokens you may want to use (list of strings). Log an error if used while not having been set.

Ids of all the additional special tokens in the vocabulary (list of integers). Log an error if used while not having been set.

List the vocabulary indices of the special tokens ('<unk>', '<cls>'...) mapped to class attributes (cls_token, unk_token...).

List all the special tokens ('<unk>', '<cls>'...) mapped to class attributes (cls_token, unk_token...).

Beginning of sentence token (string). Log an error if used while not having been set.

Id of the beginning of sentence token in the vocabulary. Log an error if used while not having been set.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

Classification token (string). E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an err

Id of the classification token in the vocabulary. E.g. to extract a summary of an input sequence leveraging self-attention along the full depth of th

batch should be a list of (sequence, target, length) tuples... Returns a padded tensor of sequences sorted from longest to shortest,

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (string) in a single string.

Converts a sequence of tokens (strings for sub-words) in a single string.

Converts a sequence of tokens (string) in a single string.

sents : a list of sentences, each a list of tokenized symbols

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. A RoBERTa sequence pair mask has the

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An XLM sequence pair mask has the fol

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the fol

Creates a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the fol

End of sentence token (string). Log an error if used while not having been set.

Id of the end of sentence token in the vocabulary. Log an error if used while not having been set.

Return the url and etag (which may be ``None``) stored for `filename`. Raise ``EnvironmentError`` if `filename` or its stored metadata do not

Parameters ---------- input_ids: torch.tensor(bs, max_seq_length) The token ids to embed. Outputs

Parameters ---------- query: torch.tensor(bs, seq_length, dim) key: torch.tensor(bs, seq_length, dim) value:

Parameters ---------- x: torch.tensor(bs, seq_length, dim) attn_mask: torch.tensor(bs, seq_length) Outputs

Parameters ---------- x: torch.tensor(bs, seq_length, dim) Input sequence embedded. attn_mask: torch.tens

Args: **p_mask**: (`optional`) ``torch.FloatTensor`` of shape `(batch_size, seq_len)` invalid position mask such as query

Args: One of ``start_states``, ``start_positions`` should be not None. If both are set, ``start_positions`` overrides ``start

Args: One of ``start_states``, ``start_positions`` should be not None. If both are set, ``start_positions`` overrides

hidden_states: float Tensor in shape [bsz, ..., seq_len, hidden_size], the hidden-states of the last layer. cls_index: [optional] positio

Self-attention (if kv is None) or attention over source sentence (provided by kv).

Compute the loss, and optionally the scores.