Convert to numpy and return a sample consumed by the batch producer.

Convert to numpy and return a sample consumed by the batch producer.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed. TODO: Design modular interface to reuse this function. This is getting

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Build token types and paddings, trim if needed, and pad if needed.

Build train, valid, and test datasets. Args: data_prefix (list[str] | str): paths to dataset (ex: ['dataset/ja_wiki_text', 'd

Build training sample. Arguments: sample: A list of sentences in which each sentence is a list token ids. target_seq_length: Desi

Biuld training sample. Arguments: sample: A list of sentences in which each sentence is a list token ids. target_seq_length: Desi

Returns list of utf-8 byte and a corresponding list of unicode strings. The reversible bpe codes work on unicode strings. This means you

Calculate l2 norm of parameters

Calculate correct over total answers

Ensure fixed arguments for a model are the same for the input arguments and the one retrieved from checkpoint.

Clips gradient norm of an iterable of parameters whose gradients are in fp32. This is adapted from torch.nn.utils.clip_grad.clip_grad_norm

Compile helper function ar runtime. Make sure this is invoked on a single process.

Merge segments A and B, add [CLS] and [SEP] and build tokentypes.

Embed a batch of tokens using the block model

Embed a batch of tokens using the query model

Build filename's path if it does not already exists.

Evaluation.

Evaluation.

Evaluate and print results on screen.

Convert fp16/bf16 `val` to fp32

Forward step.

Opposite of above function, gather values from model parallel ranks.

OpenAI's gelu implementation.

Divide sample into a and b segments.

Build the batch.

Build the batch.

Generate a batch

Build the batch.

Generate batch from context tokens.

Get samples mapping for a dataset over fixed size blocks. This function also requires a dataset of the titles for the source documents since their

Build indexed dataset.

Build the learning rate scheduler.

The output should be a cuda tensor of size 1.

Learning rate decay functions from: https://openreview.net/pdf?id=BJYwwY9ll pg. 4

Based on evaluation metric set the parallel-output flag and return the model provider.

Specifically one epoch to be used in an indexing job.

Data loader. Note that batch-size is the local (per GPU) batch-size. NOTE: This dataloader is not distributed !!!

Return a numpy array of type np.dtype giving the byte offsets. Multiplies values in the sizes array by elemsize (bytes), and then computes an

Initialize the state from a pretrained BERT model on iteration zero of ICT pretraining