Adds whitespace around any CJK character.

Train the model.

Train the model.

Write arguments to tensorboard.

Provide function that calculates accuracies.

Add the embedding of each block to the underlying FAISS index

Add the embedding of each block to the underlying FAISS index

Backward step.

Convert to numpy and return a sample consumed by the batch producer.

Convert to numpy and return a sample consumed by the batch producer.

Convert to numpy and return a sample consumed by the batch producer.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Evaluates answers presence in the set of documents. This function is supposed to be used with a large collection of documents and results. It

Ensure fixed arguments for a model are the same for the input arguments and the one retreived frm checkpoint.

Clips fp32 master gradients via ``torch.nn.utils.clip_grad_norm``. Args: max_norm (float or int): max norm of the gradie

Converts a module's immediate parameters and buffers to dtype.

Converts a network's parameters and buffers to dtype.

Converts a sequence of tokens (string) in a single string.

iterates through data and handles wrap around

Returns a list of named-entity-recognition tags of each token. Returns None if this annotation was not included.

Evaluation.

Convert fp16 `val` to fp32

Convert fp32 `val` to fp16

Opposite of above function, gather values from model parallel ranks.

OpenAI's gelu implementation.

Return my rank for the data parallel group.

Build the learning rate scheduler.

Learning rate decay functions from: https://openreview.net/pdf?id=BJYwwY9ll pg. 4

Data loader. Note that batch-size is the local (per GPU) batch-size. NOTE: This dataloader is not distributed !!!

Set up the optimizer.

Divide params into with-weight-decay and without-weight-decay groups. Layernorms and baises will have no weight decay but the rest will.

Return T0 model.

Return T0 model.

Check if a document contains an answer string. If `match_type` is string, token matching is done between the text and answer. If `match_type`

Initialize torch.distributed and mpu.

Initialize the state from pre-trained DPR model and pre-trained T5 mode on iteration zero of pretraining

Whether the current buffer hold on to any memory.

Loads a vocabulary file into a dictionary.