Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary.

Send all values to device by calling v.to(device)

Initialize and prunes weights if needed.

Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2) Basically works like a Linear layer but the weights are

Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch. Padding side

Add a new hypothesis to the list.

In distributed setting, this function only download checkpoint at local rank 0.

Prune a linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True.

Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) an

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-wo

Updates attributes of this class with attributes from `config_dict`. Args: :obj:`Dict[str, any]`: Dictionary of

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

Performs all_gather operation on the provided tensors. *** Warning ***: torch.distributed.all_gather has no gradient.

Serialize an object. Returns: Implementation-dependent bytes-like object

Returns the model's input embeddings. Returns: :obj:`nn.Module`: A torch module mapping vocabulary to hi

Resolve a model identifier, and a file name, to a HF-hosted url on either S3 or Cloudfront (a Content Delivery Network, or CDN). Cloudfr

Args: buf: the output of `dumps`.

Args: buf: the output of `dumps`.

Tie the weights between the input embeddings and the output embeddings. If the `torchscript` flag is set in the configuration, can't

Serializes this instance to a Python dictionary. Returns: :obj:`Dict[str, any]`: Dictionary of all the attributes that m

Returns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input s

Returns the tokenized representation of given input string(s) Parameters ---------- texts : Union[str, List[str]] An input s

Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args:

Runs basic whitespace cleaning and splitting on a piece of text.

Pad encoded inputs (on left/right and up to predefined legnth or max length in the batch) Args: encoded_inputs: Dictionary of to

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-wo

During generation, decide whether to pass the `past` variable to the next forward pass.

Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you

Converts a single index or a sequence of indices (integers) in a token " (resp.) a sequence of tokens (str), using the vocabulary and add

Converts a sequence of tokens (string) in a single string. The most simple way to do it is ' '.join(self.convert_ids_to_tokens(token_ids)

Compute the DICE loss, similar to generalized IOU for masks Args: inputs: A float tensor of arbitrary shape. The pred

Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing ``self.convert_tokens_to_ids(sel

Returns a dictionary containing the encoded sequence or sequence pair and additional information: the mask for sequence classificatio

r"""Instantiate a pretrained pytorch model from a pre-trained model configuration. The model is set in evaluation mode by default using ``mod

Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings).

Initialize the weights in backbone. Args: pretrained (str, optional): Path to pre-trained weights. Defaults to No

L2-normalize columns of X

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates

setup logger for training and testing. Args: save_dir(str): location to save log file distributed_rank(int): device rank when mult

Serializes this instance to a JSON string. Args: use_diff (:obj:`bool`): If set to True, only the differ

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (batch size,

Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C)

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates

Checks that `replacement`, which is intended to replace `original` is of the right type. The type is correct if it matches exactly or is one of a

-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]

Decodes a raw config value (e.g., from a yaml config files or command line argument) into a Python object.

Generate sequences for each example with beam search.

Generate sequences for each example without beam search (num_beams == 1). All returned sequence are generated independantly.

Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at the en

Initialize the weights

Checks whether CP is the codepoint of a CJK character.