Creates a `truncated_normal_initializer` with the given range.

Returns a list of the shape of tensor, preferring static dimensions. Args: tensor: A tf.Tensor object to find the shape of. expected_rank:

Converts a sequence of [tokens|ids] using the vocab.

Perform dropout. Args: input_tensor: float Tensor. dropout_prob: Python float. The probability of dropping out a value (NOT of *keepi

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

Convert a set of `InputExample`s to a TFRecord file.

Creates an `input_fn` closure to be passed to TPUEstimator.

See base class.

Run layer normalization on the last dimension of the tensor.

Reshapes a >= rank 2 tensor to a rank 2 tensor (i.e., a matrix).

Runs basic whitespace cleaning and splitting on a piece of text.

See base class.

Reshapes a rank 2 tensor back to its original rank >= 2 tensor.

Performs invalid character removal and whitespace cleanup on text.

Decodes a record to a TensorFlow example.

Whether to use L2 weight decay for `param_name`.

Get the variable name from the tensor name.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

See base class.

Raises an exception if the tensor rank is not of the expected rank. Args: tensor: A tf.Tensor to check the rank of. expected_rank: Python i

Performs multi-headed attention from `from_tensor` to `to_tensor`. This is an implementation of multi-headed attention based on "Attention is all

Converts a single `InputExample` into a single `InputFeatures`.

Create 3D attention mask from a 2D tensor mask. Args: from_tensor: 2D or 3D Tensor of shape [batch_size, from_seq_length, ...]. to_mask: in

Looks up words embeddings for id tensor. Args: input_ids: int32 Tensor of shape [batch_size, seq_length] containing word ids. vocab_s

Performs various post-processing on a word embedding tensor. Args: input_tensor: float Tensor of shape [batch_size, seq_length, embedding

Constructs a `BertConfig` from a Python dictionary of parameters.

Constructs a `BertConfig` from a json file of parameters.

Gathers the vectors at the specific positions over a minibatch.

Maps a string to a Python function, e.g., "relu" => `tf.nn.relu`. Args: activation_string: String name of the activation function. Returns:

Get loss and log probs for the masked LM.

Gets final hidden layer of encoder. Returns: float Tensor of shape [batch_size, seq_length, hidden_size] corresponding to the final h

See base class.

Runs layer normalization followed by dropout.

Loads a vocabulary file into a dictionary.

Computes the loss and accuracy of the model.

Returns `model_fn` closure for TPUEstimator.

Serializes this instance to a Python dictionary.

Multi-headed, multi-layer Transformer from "Attention is All You Need". This is almost an exact implementation of the original Transformer encoder.

Constructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input.

Constructs a AdamWeightDecayOptimizer.

Constructs BertConfig. Args: vocab_size: Vocabulary size of `inputs_ids` in `BertModel`. hidden_size: Size of the encoder layers and

Constructor for BertModel. Args: config: `BertConfig` instance. is_training: bool. true for training model, false for eval model. Con

Constructs a InputExample. Args: guid: Unique id for the example. text_a: string. The untokenized text of the first sequence. For sin

Reads a tab separated value file.

Creates an optimizer training op.

Gaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 Args: x: float Tensor

Compute the union of the current variables and checkpoint variables.

Gets output of the embedding lookup (i.e., input to the transformer). Returns: float Tensor of shape [batch_size, seq_length, hidden_size]

Gets the list of labels for this data set.

Gets a collection of `InputExample`s for the train set.

The actual input function.

The `model_fn` for TPUEstimator.

Returns text encoded in a way suitable for print or `tf.logging`.

Serializes this instance to a JSON string.

Tokenizes a piece of text.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given voca

Checks whether the casing config is consistent with the checkpoint name.