Index sentences with a dictionary.

Initialize AMP optimizer.

If there are enough hypotheses and that none of the hypotheses being generated can become better than the worst one in the heap, then

Reload pretrained embeddings from a fastText binary file.

Load pair regression/classification bi-sentence tasks

Load XNLI cross-lingual classification data.

Load pair regression/classification bi-sentence tasks

Load a binarized fastText model.

Loads the saved model from disk if the directory exists. Otherwise it will download the model and tokenizer from hugging face. Returns

Load monolingual data.

Load parallel data.

Decide of random words to mask out, and what target they get assigned.

Decide of random words to mask out. We specify the random generator to ensure that the test is the same at each epoch.

Limit the vocabulary size.

Threshold on the word frequency counts.

Normalize unicode underlying representation

Parser.

Parser.

Parse the configuration of lambda coefficient (for scheduling). x = "3" # lambda will be a constant equal to x x = "0:1,

Given the last hidden state, compute word scores and/or the loss. `pred_mask` is a ByteTensor of shape (slen, bs), filled with 1

Print statistics about the training.

Process a binarized dataset and log main statistics.

Remove URLs in text

Reload pretrained embeddings from a text file.

Create a dictionary from a vocabulary file.

Reload a checkpoint if we find one.

Remove empty sentences.

Remove sentences exceeding a certain length.

Remove sentences exceeding a certain length.

Remove multiple spaces

Save clean txt files to json format

Clean txt files in indir and save clean file to output directory

Only select a subset of the dataset.

Set optimizers.

Set parameters.

Split text into 1 sentence per line

Train the model

Finetune for one epoch on the XNLI English training set.

Finetune for one epoch on the training set.

Update all lambda coefficients.

Randomly blank input words.

Randomly drop input words.

Randomly shuffle input words.

Returns whether a word is in the dictionary.

Compare this dictionary with another one.

Returns the word of the specified index.

Transformer model (encoder or decoder).

Initialize n-best list of hypotheses.

Wrapper on top of the different sentence embedders. Returns sequence-wise or single-vector sentence representations.

Prepare batches for data iterator.

Build language model evaluator.

Build encoder / decoder evaluator.

Initialize GLUE trainer / evaluator. Initial `embedder` should be on CPU to save memory.

Initialize XNLI trainer / evaluator. Initial `embedder` should be on CPU to save memory.

Number of hypotheses in the list.

Returns the number of words in the dictionary.

Number of sentences in the dataset.

Number of sentences in the dataset.

Number of sentences in the dataset.

Generate scores and indices given keys and unnormalized queries.

Parse boolean arguments from the command line.

Back-translation step for machine translation.