Initialize the state from a pretrained BERT model on iteration zero of ICT pretraining

Whether the current buffer hold on to any memory.

Load the necessary attributes: model, dataloader and empty BlockData

Populate members from instance saved to file

selectively load ICT models for indexing/retrieving from ICT or REALM checkpoints

load sentencepiece model and parse vocab

Loads a vocabulary file into a dictionary.

Loads a vocabulary file into a dictionary.

Main program.

Create a BlockData data structure by running an IndexBuilder over an ICT Dataset - Include all args needed for initial model specification Ot

Main program.

Main program.

Make lazy version of `data_type` field of the file. Byte offsets corresponding to data indices are stored in a `.len.pkl` data file.

makes training/val/test

Load train/val/test dataset from shuffled TFRecords

Helper function to instantiate a tokenizer given common combinations of options.

helper function to mask `idx` token from `tokens` according to section 3.3.1 of https://arxiv.org/pdf/1810.04805.pdf

Combine all the shards made using self.save_shard()

Copy model gradients to master gradients. Args: model_params: List of model parameters created by :func:`prep_param_lists`.

package multiple files

Pad sequences and convert them to numpy.

Parse all arguments.

Process batch and produce inputs for the model.

Abstract method that takes a single path / filename and returns a list of dataset samples, each sample being a dict of {'text_a':

Read in RACE files, combine, clean-up, tokenize, and convert to samples.

Replace fp16 linear with quantized linear

Recursively lint all python files in chosen subdirectories of megatron-lm

Simple GPU memory report.

Reset timer.

Check ETag on S3 object.

Pull a file directly from S3.

Save the block data that was created this in this process

Set args, tokenizer, tensorboard-writer, adlr-autoresume, and timers.

Set model parallel rank.

Set the model parallel size

Add a list of additional tokens to the encoder. The additional tokens are indexed starting from the last index of the current