Beginning of sentence token id

Id of the beginning of sentence token in the vocabulary.

Broadcast data from rank zero of each model parallel group to the members of the same model parallel group. Arguments: keys: list of

Check for autoresume signal and exit if it is received.

Search through all the top docs to see if they have any of the answers.

Checkpoint a model or part of the model. This has been directly copied from torch.utils.checkpoint.

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.

Clips gradient norm of an iterable of parameters. This is adapted from torch.nn.utils.clip_grad.clip_grad_norm_ and added functionality to ha

Set the groups to none.

Group consecutive entity tokens with the same NER tag.

End of sentence token id

Id of the end of sentence token in the vocabulary.

Helper function to evaluate and dump results on screen.

Run a forward pass for each of the models and return the respective embeddings.

Return the data currently in use.

Return world size for the data parallel group.

Build masks and position id for left to right model.

Get the memory buffer.

Provide extra arguments required for tasks.

Initializ the memory buffer for the checkpointed activations.

Initialize model data parallel groups. Arguments: model_parallel_size: number of GPUs used to parallelize model. Let's say we h

When running with :class:`FP16_Optimizer`, ``.grad`` attributes of a model's fp16 leaves should not be regarded as truthful,

Dictionary from vocab id token to text token.

Useful for code segments that may be accessed with or without mpu initialization