Returns a list of the shape of tensor, preferring static dimensions. Args: tensor: A tf.Tensor object to find the shape of. expected_

Creates a `truncated_normal_initializer` with the given range.

Converts a sequence of [tokens|ids] using the vocab.

:param x_flat: [batch_size*seq_length, width] :return: A fixed up tensor of size [batch_size, num_attention_heads, seq_length, size_per_head]

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

Run layer normalization on the last dimension of the tensor.

Runs basic whitespace cleaning and splitting on a piece of text.

Decodes a record to a TensorFlow example.

Does top-p sampling. if ignore_ids is on, then we will zero out those logits. :param logits: [batch_size, vocab_size] tensor :param ignor

Perform dropout. Args: input_tensor: float Tensor. dropout_prob: Python float. The probability of dropping out a value (NOT of

Constructs a `NewsConfig` from a json file of parameters.

Helper function that samples from grover for a single step :param tokens: [batch_size, n_ctx_b] tokens that we will predict from :param i

Performs invalid character removal and whitespace cleanup on text.

Whether to use L2 weight decay for `param_name`.

Get the variable name from the tensor name.

Checks whether CP is the codepoint of a CJK character.

Checks whether `chars` is a control character.

Checks whether `chars` is a punctuation character.

Checks whether `chars` is a whitespace character.

Splits punctuation on a piece of text.

Strips accents from a piece of text.

Adds whitespace around any CJK character.

Does top-k sampling. if ignore_ids is on, then we will zero out those logits. :param logits: [batch_size, vocab_size] tensor :param ignor

See base class.

Iterate through the provided filename + tokenize

Raises an exception if the tensor rank is not of the expected rank. Args: tensor: A tf.Tensor to check the rank of. expected_rank: Py

:param x_flat: Tensor input, should be [batch_size*seq_length, dim] :param attention_mask: Attention mask to use of size [seq_length, seq_le

We apply a sliding window to fix long sequences, and use a buffer that combines short sequences.

reur and position embeddings :param input_ids: int Tensor of shape [batch_size, seq_length]. :param vocab_size: number of words in vocab :

Given some tokens that were generated, extract the target :param output_tokens: [num_tokens] thing that was generated :param encoder: how

Constructs a `NewsConfig` from a Python dictionary of parameters.

Compute the union of the current variables and checkpoint variables.

this is a TPU compatible version of tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd) where the lower right triangle contains 1s

same signature as sample_step

Creates an `input_fn` closure to be passed to TPUEstimator.

:return: stuff

Loads a vocabulary file into a dictionary.

Mask attention so that we're only predicting going forward :param attention_scores: [batch, heads, dst_sequence, src_sequence], where informa

Returns `model_fn` closure for TPUEstimator.

:param x: The attention output. It should be [batch_size*seq_length, dim] :param intermediate_size: the hidden projection. By default this is

V1 version of: sample outputs from a model, and do it all at once :param news_config: Configuration used to construct the model :param in

Serializes this instance to a Python dictionary.

Constructs a BasicTokenizer. Args: do_lower_case: Whether to lower case the input.

Constructs NewsConfig. Args: vocab_size: Vocabulary size of `inputs_ids` in `GroverModel`. hidden_size: Size of the layer

:param config: :param is_training: :param input_ids: Tensor thats of size [batch_size, seq_length] :param cache: Opti

Constructs a AdaFactorOptimizer.

Estimate the scale of the parameters from the current values. We include a minimum value of 0.001 to give it a chance to escape 0 if i

for whatever reason this didn't work when I ran it on more than one at once... ugh.

Convert a set of `InputExample`s to a TFRecord file.

Creates an `input_fn` closure to be passed to TPUEstimator.

Construct a host call to log scalars when training on TPU. Args: metric_dict: A dict of the tensors to be logged. model_dir: The loca

Creates an optimizer training op.

Gaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 Args: input_t

Training host call. Creates scalar summaries for training metrics. This function is executed on the CPU and should not directly reference

The actual input function.

The `model_fn` for TPUEstimator.

Extract pooled output given a token that says where we should look :param clf_token: :return:

Returns text encoded in a way suitable for print or `tf.logging`.

Serializes this instance to a JSON string.

Tokenizes a piece of text.

Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given voca

Checks whether the casing config is consistent with the checkpoint name.