Build model.

Check datasets parameters.

Check models parameters.

Next word prediction step (causal prediction). CLM objective.

End the epoch.

Compute the loss, and optionally the scores.

Self-attention (if kv is None) or attention over source sentence (provided by kv).

Read from the memory.

Generate queries from hidden states by either repeating them or creating some shuffled version.

Compute queries using either grouped 1D convolutions or ModuleList + concat.

GELU activation https://arxiv.org/abs/1606.08415 https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_pytorch.p

Generate random Gaussian keys.

Generate scores and indices given unnormalized queries.

Generate scores and indices given unnormalized queries.

Return a sentences iterator.

Return a sentences iterator.

Return a sentences iterator.

Input: - matrix of size (m, d) (keys) - matrix of size (n, d) (queries) - number of nearest neighbors - distance

Generate random uniform keys (same initialization as nn.Linear).

Handle single and multi-GPU / multi-node / SLURM jobs. Initialize the following variables: - n_nodes - node_id - loca

Initialize keys.

Initialize keys.

Handle signals sent by SLURM for time limit / pre-emption.

Initialize the experience: - dump parameters - create a logger

End of iteration.

Load monolingual data. The returned dictionary contains: - dico (dictionary) - vocab (FloatTensor) - train / valid /

Masked word prediction step. MLM objective is lang2 is None, TLM objective otherwise.

Machine translation step. Can also be used for denoising auto-encoding.

Parallel classification step. Predict if pairs of sentences are mutual translations of each other.

Keep only characters belonging to the character set of a language Args: -- text: input text -- code page: for each language. Ex

Register memory parameters

Create a sentence embedder from a pretrained model.

Run GLUE training / evaluation.

Run XNLI training / evaluation.

Run training/evaluation

Run all evaluations.

Save best models according to given validation metrics.

Save the models periodically.

Only select a subset of the dataset.

Set the probability of sampling specific languages / language pairs during training.

projection

Randomize training order.

compute context