Customized load.

LM logits using word embedding weights.

For easy load.

Make sure the input variable is not None.

Gather tensors and concatinate along the last dimension.

Initialize affine weight for model parallel. Build the master weight on all processes and scatter the relevant chunk.

Initialize affine weight for model parallel on GPU.

Check whether it is rank 0. For AML, check if it is rank 0 of a node

All-reduce the the input tensor across model parallel group.

Create a Faiss Flat index with inner product as the metric to search against

Split the tensor along its last dimension and keep the corresponding slice.

Zero out the gradient for a group of parameters. Note: copied from torch.optim.optimizer.

Build the model.

Buld dataloader given an input dataset.

Clear the embedding data structures to save memory. The metadata ends up getting used, and is also much smaller in dimensiona

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

Convert fp32 `val` to fp16/bf16

Return my rank for the data parallel group.

Return tensorboard writer. It can be None so no need to check if it is initialized.

Returns a 2-dimensional (2-D) attention mask :param source_block: 1-D array :param target_block: 1-D array

Runs basic whitespace cleaning and splitting on a piece of text.

Initialize tokenizer.

Build train, valid, and test datasets.

Use multi-tensor-applier to copy values from one list to another. We don't have a blfoat16 implementation so for now if the overflow_buf is no

Allocate a memory buffer.

Helper method to run backward step with model split into chunks (run set_virtual_pipeline_model_parallel_rank() before calling backwar

Convert to numpy and return a sample consumed by the batch producer.

Build token types and paddings, trim if needed, and pad if needed.

XXX

Check for autoresume signal and exit if it is received.

Apply conversion to val. Recursively apply conversion if `val` #is a nested tuple/list structure.

Converts a sequence of ids in BPE tokens using the vocab.

Converts a sequence of tokens into ids using the vocab.

Creates the predictions for the masked LM objective. Note: Tokens here are vocab ids and not text tokens.