Returns the index of the specified word.

Move tensors to CUDA.

Finetune for one epoch on the training set.

Step.

Concat batches with different languages.

Truncate long sentences.

Evaluate on XNLI validation and test sets, for all languages.

Load a binarized dataset.

Update dictionary parameters.

Parse optimizer parameters. Input should be of the form: - "sgd,lr=0.01" - "adagrad,lr=0.1,lr_decay=0.05"

Return a batch of sentences from a dataset.

Inputs: `x` : LongTensor of shape (slen, bs) `lengths` : LongTensor of shape (bs,) Outputs:

Check that the dictionary is valid.

Clean up an input text

Optimize.

Sanity checks.

Sanity checks.

Get a list of valid book ids from an input directory Returns: list of book ids

Decode a sentence given initial start. `x`: - LongTensor(bs, slen) <EOS> W1 W2 W3 <EOS> <PAD>

Create a new iterator for a dataset.

Take as input a list of n sentences (torch.LongTensor vectors) and return a tensor of size (slen, n) where slen is the length of the

Convert a batch of sentences to a list of text sentences.

Generate a convolutional neural network.

Evaluate memory usage (HashingMemory / FFN).

Generate a feedforward neural network.

Take a file segmented with BPE and restore it to its original segmentation.

For float16 only. Sub-sample sentences in a batch, and add padding, so that each dimension is a multiple of 8.

Save the model / checkpoints.

Pretrain word embeddings.

Initialize trainer.

Initialize evaluator.

Generate scores and indices given keys and unnormalized queries.

Generate scores and indices given keys and unnormalized queries.

Add noise to the encoder input.

Compute the batched cartesian product between two matrices. Input: a: Tensor(n, d1) b: Tensor(n, d2) Output: outp

Create a logger. Use a different log file for each process.

Evaluate on XNLI validation and test sets, for all languages.

Recursively find all instances of a specific module inside a module.

Forward function with different forward modes. ### Small hack to handle PyTorch distributed.

Prepare a batch (for causal or non-causal mode).

Create a new iterator for a dataset.

Build data iterator.

Get a monolingual data iterator.

Build data iterator.

`metric` can be faiss.METRIC_INNER_PRODUCT or faiss.METRIC_L2 https://github.com/facebookresearch/faiss/blob/master/gpu/test/test_pytorch_fai

Compute scores.

Returns corresponding url to download from an input book id Args: ::book_id:: book ID number ::encoding:: encoding type pre

Reload pretrained embeddings.

Prepare masked tokens inputs/labels for masked language modeling: 80% MASK, 10% random, 10% original.

Remove empty sentences.

Only select a subset of the dataset.

Remove lines that are part of the Project Gutenberg header or footer. Note: this function is a port of the C++ utility by Johannes Krugel. Th

Add a new hypothesis to the list.

Check and initialize memory parameters.

Check if an url exists

Converts `text` to Unicode (if it's not already), assuming UTF-8 input.

This function creates keys and returns them. I guess you could see that from the name of the function and the fact that is has a retu

Create reference files for BLEU evaluation.

Download books from a list of book ids and save all to output_dir

Evaluate on XNLI validation and test sets, for all languages.

Given a file of hypothesis and reference files, evaluate the BLEU score using Moses scripts.

Evaluate perplexity and next word prediction accuracy.

Evaluate perplexity and next word prediction accuracy.

Evaluate perplexity and next word prediction accuracy.

Inputs: `x` LongTensor(slen, bs), containing word indices `lengths` LongTensor(bs), containing the length of each sen

Decode a sentence given initial start. `x`: - LongTensor(bs, slen) <EOS> W1 W2 W3 <EOS> <PAD>

Return a sentences iterator, given the associated sentence batches.

Return a sentences iterator, given the associated sentence batches.

Return a list of unique book ids from a list of book urls

Returns a list of unique book ids

Create a directory to store the experiment.

Compute a lambda value according to its schedule configuration.

Generate hidden states mask, and optionally an attention mask.

Generate slices of hidden dimensions. Used when there are multiple heads and/or different set of keys, and that there is no query network

Returns subdirectory that a book_id will be found in a gutenberg mirror.