Creates `TrainingInstance`s for a single document.

Creates the predictions for the masked LM objective.

Creates an optimizer training op.

Creates an optimizer training op.

Looks up words embeddings for id tensor. Args: input_ids: int32 Tensor of shape [batch_size, seq_length] containing word ids. vocab_s

Gathers the vectors at the specific positions over a minibatch.

Gaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 Args: x: float Tensor

Compute the union of the current variables and checkpoint variables.

Gets a collection of `InputExample`s for the dev set.

Gets output of the embedding lookup (i.e., input to the transformer). Returns: float Tensor of shape [batch_size, seq_length, hidden_size]

Gets the list of labels for this data set.

Gets a collection of `InputExample`s for prediction.

Gets a collection of `InputExample`s for the train set.

The actual input function.

The actual input function.

Creates an `input_fn` closure to be passed to TPUEstimator.

Run layer normalization on the last dimension of the tensor.

Computes the loss and accuracy of the model.

The `model_fn` for TPUEstimator.

The `model_fn` for TPUEstimator.

Multi-class precision metric for Tensorflow Parameters ---------- labels : Tensor of tf.int32 or tf.int64 The true labels

Returns text encoded in a way suitable for print or `tf.logging`.

Multi-class recall metric for Tensorflow Parameters ---------- labels : Tensor of tf.int32 or tf.int64 The true labels pr

Reshapes a rank 2 tensor back to its original rank >= 2 tensor.

Reshapes a >= rank 2 tensor to a rank 2 tensor (i.e., a matrix).

test of Factorized embedding parameterization :return:

test of share parameters across all layers: how many parameter after share parameter across layers of transformer. :return:

Serializes this instance to a JSON string.

Tokenizes a piece of text.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given voca

Checks whether the casing config is consistent with the checkpoint name.