Search through all the top docs to see if they have any of the answers.

Checkpoint a model or part of the model. This has been directly copied from torch.utils.checkpoint.

Clean up a list of simple English tokenization artifacts like spaces before punctuations and abreviated forms.

Clips gradient norm of an iterable of parameters. This is adapted from torch.nn.utils.clip_grad.clip_grad_norm_ and added functionality to ha

Set the groups to none.

Group consecutive entity tokens with the same NER tag.

End of sentence token id

Id of the end of sentence token in the vocabulary.

Helper function to evaluate and dump results on screen.

Run a forward pass for each of the models and return the respective embeddings.

Return the data currently in use.

Return world size for the data parallel group.

Build masks and position id for left to right model.

Get the memory buffer.

Provide extra arguments required for tasks.

Initializ the memory buffer for the checkpointed activations.

Initialize the state from a pretrained BERT model on iteration zero of ICT pretraining

Initialize model data parallel groups. Arguments: model_parallel_size: number of GPUs used to parallelize model. Let's say we h

When running with :class:`FP16_Optimizer`, ``.grad`` attributes of a model's fp16 leaves should not be regarded as truthful,

Dictionary from vocab id token to text token.

Useful for code segments that may be accessed with or without mpu initialization

Efficient L2 norm gradient clipping.