Create a 1D, 2D, or 3D convolution module.

Create a linear module.

Make a standard normalization layer. :param channels: number of input channels. :return: an nn.Module for normalization.

Zero out the parameters of a module and return it.

Converts a single token, or a sequence of tokens, (str/unicode) in a single integer id (resp. a sequence of ids), using the vocabulary.

Create sinusoidal timestep embeddings. :param timesteps: a 1-D Tensor of N indices, one per batch element. These may be

Initialize and prunes weights if needed.

Conv1D layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2) Basically works like a Linear layer but the weights are

Evaluate a function without caching intermediate activations, allowing for reduced memory at the expense of extra compute in the backward pas

sample from posterior Gaussian family Args: mu: Tensor Mean of gaussian distribution with shape (batch, nz)

Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at the en

Initialize and prunes weights if needed.

Prune a linear layer (a model parameters) to keep only entries in index. Return the pruned layer as a new layer with requires_grad=True.

Tie or clone module weights depending of weither we are using TorchScript or not

compute log p(z,x) Args: x: Tensor input with shape [batch, seq_len] z: Tensor evaluat

Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args:

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-wo

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing ``self.convert_tokens_to

Prune a Conv1D layer (a model parameters) to keep only entries in index. A Conv1D work as a Linear layer (see e.g. BERT) but the weights are

Make sure we are sharing the input and output embeddings. Export to TorchScript can't handle parameter sharing so we are cloning them ins

Runs basic whitespace cleaning and splitting on a piece of text.

Initialize the weights.

Tie or clone module weights depending of weither we are using TorchScript or not

Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-wo

Add a dictionary of special tokens (eos, pad, cls...) to the encoder and link them to class attributes. If special tokens are NOT in

Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices start

context_info_list: [[context_type, context, ratio]] for 'attention'

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.

Converts a single index or a sequence of indices (integers) in a token " (resp.) a sequence of tokens (str/unicode), using the vocabulary

Converts a sequence of tokens (string) in a single string. The most simple way to do it is ' '.join(self.convert_ids_to_tokens(token_ids)

A counter for the `thop` package to count the operations in an attention operation. Meant to be used like: macs, params = thop.pr