Runs basic whitespace cleaning and splitting on a piece of text.

Given something that might be a URL (or might be a local path), determine which. If it's a URL, download the file and cache it, and retur

Converts a sequence of tokens into ids using the vocab.

Creates examples for the training and dev sets.

Constructs a `BertConfig` from a json file of parameters.

Split a full s3 path into the bucket name and path.

Serializes this instance to a JSON string.

Performs invalid character removal and whitespace cleanup on text.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

Creates the predictions for the masked LM objective. This is mostly copied from the Google BERT repo, but with several refactors to clean it up a

Constructs a `BertConfig` from a Python dictionary of parameters.

Given a URL, look for the corresponding dataset in the local cache. If it's not there, download it. Then return the path to the cached file.

:param progress: value between 0 and 1 (unless going beyond t_total steps) specifying training progress :return: learni

Load tf checkpoints in a pytorch model

Loads a vocabulary file into a dictionary.

Check ETag on S3 object.

Pull a file directly from S3.

Performs a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model

Serializes this instance to a Python dictionary.

Convert `url` into a hashed filename in a repeatable way. If `etag` is specified, append its hash to the url's, delimited by a period.

Constructs a BertTokenizer. Args: vocab_file: Path to a one-wordpiece-per-line vocabulary file do_lower_case: Whether to

Constructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input.

:param warmup: what fraction of t_total steps will be used for linear warmup :param t_total: how many training steps (updates) are p

:param warmup: see LRSchedule :param t_total: see LRSchedule :param cycles: number of cycles. Default: 0.5, cor

Constructs BertConfig. Args: vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `BertModel`. hidden_s

Construct a layernorm module in the TF style (epsilon inside the square root).

Converts a sequence of ids in wordpiece tokens using the vocab.

Return the url and etag (which may be ``None``) stored for `filename`. Raise ``EnvironmentError`` if `filename` or its stored metadata do not

Instantiate a PreTrainedBertModel from a pre-trained model file. Download and cache the pre-trained model file if needed.

Instantiate a BertPreTrainedModel from a pre-trained model file or a pytorch state dict. Download and cache the pre-trained model fil

Implementation of the gelu activation function. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results

:param step: which of t_total steps we're on :param nowarn: set to True to suppress warning regarding training beyond specified '