Take corpus, split it into sentences, and extract word frequencies. Write frequencies to `filepath` as a tsv. Only write the first MAX

Return my rank for the data parallel group.

Get a collection of `InputExample`s for the dev set.

Given a URL, look for the corresponding dataset in the local cache. If it's not there, download it. Then return the path to the cached file.

Build the learning rate scheduler.

Set up the optimizer.

Get dataset splits from comma separated string list

Load the data on rank zero and boradcast number of tokens to all GPUS.

get type token corresponding to `name`

Get a collection of `InputExample`s for the unlabeled set.

If there are enough hypotheses and that none of the hypotheses being generated can become better than the worst one in the heap, then

load sentencepiece model and parse vocab

Loads a vocabulary file into a dictionary.

Main training program.

Main training program.

makes training/val/test

Load train/val/test dataset from shuffled TFRecords

Helper function to instantiate a tokenizer given common combinations of options.

helper function to mask `idx` token from `tokens` according to section 3.3.1 of https://arxiv.org/pdf/1810.04805.pdf

Copy model gradients to master gradients. Args: model_params: List of model parameters created by :func:`prep_param_lists`.

Simple GPU memory report.

Check ETag on S3 object.

Pull a file directly from S3.

Save a model checkpoint.

Init method based on N(0, sigma/sqrt(2*num_layers).

Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current

Setup model and optimizer.