Calculate correct over total answers and return prediction if the `output_predictions` is true.

Ensure fixed arguments for a model are the same for the input arguments and the one retreived frm checkpoint.

Clips fp32 master gradients via ``torch.nn.utils.clip_grad_norm``. Args: max_norm (float or int): max norm of the gradie

Converts a sequence of ids in wordpiece tokens using the vocab.

Converts a module's immediate parameters and buffers to dtype.

Converts a network's parameters and buffers to dtype.

Converts a sequence of tokens into ids using the vocab.

Converts a sequence of tokens into ids using the vocab.

Creates the predictions for the masked LM objective. Note: Tokens here are vocab ids and not text tokens.

Mask sequence pair for BERT training according to: https://github.com/google-research/bert/blob/master/create_pretraining_data.py#L33

fetches a random sentencepair corresponding to rng state similar to https://github.com/google-research/bert/blob/master/create_pretra

Merge segments A and B, add [CLS] and [SEP] and build tokentypes.

iterates through data and handles wrap around

Embed a batch of tokens using the query model

Evaluation.

Evaluation.

Evaluate and print results on screen.

Check if we've already made a lazy version of this file for the `data_type` field.

Forward step.

Convert fp16 `val` to fp32

Convert fp32 `val` to fp16

Opposite of above function, gather values from model parallel ranks.

OpenAI's gelu implementation.

OpenAI's gelu implementation.

Divide sample into a and b segments.

Build the batch.

Generate a batch

Get samples mapping for a dataset over fixed size blocks. This function also requires a dataset of the titles for the source documents since their

Take corpus, split it into sentences, and extract word frequencies. Write frequencies to `filepath` as a tsv. Only write the first MAX_SE

Return my rank for the data parallel group.

gets path extension

Given a URL, look for the corresponding dataset in the local cache. If it's not there, download it. Then return the path to the cached file.

Get a dataset which uses block samples mappings to get ICT/block indexing data (via get_block()) rather than for training, since it is only built

Build indexed dataset.

Build the learning rate scheduler.

Learning rate decay functions from: https://openreview.net/pdf?id=BJYwwY9ll pg. 4

Based on evaluation metric set the parallel-output flag and return the model provider.

Specifically one epoch to be used in an indexing job.

Set up the optimizer.

Divide params into with-weight-decay and without-weight-decay groups. Layernorms and baises will have no weight decay but the rest will.

Get dataset splits from comma separated string list

get type token corresponding to `name`