Combine all the shards made using self.save_shard()

Combine all the shards made using self.save_shard()

Copy model gradients to master gradients. Args: model_params: List of model parameters created by :func:`prep_param_lists`.

Parse all arguments.

Abstract method that takes a filename and returns a list of dataset samples, each sample being a dict of {'text': string, 'text':

Test if a regex pattern is contained within a text.

Simple GPU memory report.

Reset timer.

Save the block data that was created this in this process

Save the block data that was created this in this process

Set model parallel rank.

Set the model parallel size

Return a view of the list of tokens from [i, j).

Split a tensor along its last dimension. Arguments: tensor: input tensor. num_partitions: number of partitions to split the tensor

Break a tensor into equal 1D chunks.

Use this function to override the state dict for saving checkpoints.

Save dict with state dicts of each of the models.

Build train and validation dataset.

Returns the original text (with whitespace reinserted).

Reload index with new data loaded from disk. Should be performed after each indexer job completes.

Delete existing index and create a new

gather tensors of different sizes along the first dimension

Zero fp32 and fp16 parameter grads.