Instantiate a pre-processed corpus.

Instantiate a GPT2PreTrainedModel from a pre-trained model file or a pytorch state dict. Download and cache the pre-trained model fil

Instantiate a BertPreTrainedModel from a pre-trained model file or a pytorch state dict. Download and cache the pre-trained model fil

Instantiate a OpenAIGPTPreTrainedModel from a pre-trained model file or a pytorch state dict. Download and cache the pre-trained mode

Implementation of the gelu activation function. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results

:param step: which of t_total steps we're on :param nowarn: set to True to suppress warning regarding training beyond specified '

Initialize the weights.

Initialize the weights.

Initialize the weights.

r""" Computes log probabilities for all :math:`n\_classes` From: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/adaptive.py

Extract a de-duped collection (set) of text from a file. Expected file format is one item per line.

Wrapper function for s3 requests in order to create more helpful error messages.

Save the tokenizer vocabulary and merge files to a directory.

Save the tokenizer vocabulary to a directory or file.

Save the tokenizer vocabulary and merge files to a directory.

Save the tokenizer vocabulary to a directory or file.

Update input and output embeddings with new embedding matrice Make sure we are sharing the embeddings

Update input and output embeddings with new embedding matrice Make sure we are sharing the embeddings

Make sure we are sharing the embeddings

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

Save this instance to a json file.

Save this instance to a json file.

Save this instance to a json file.

Save this instance to a json file.

Tokenizes a piece of text.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the gi