Whether to use L2 weight decay for `param_name`.

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

Parameters ---------- input_ids: tf.Tensor(bs, max_seq_length) The token ids to embed. Outputs -

Applies embedding based on inputs tensor.

Applies embedding based on inputs tensor.

force one of token_ids to be generated by setting prob of all other tokens to 0

Generate sequences for each example with beam search.

Generate sequences for each example without beam search (num_beams == 1). All returned sequence are generated independantly.

Finds the best answer span for the extractive Q&A model for one passage. It returns the best span by descending `span_score` order an

If TRANSFORMERS_VERBOSITY env var is set to one of the valid choices return that as the new default level. If it is not - fall back to ``_def

compute global attn indices required throughout forward pass

compute global attn indices required throughout forward pass

Computes the index of the first occurance of `sep_token_id`.

Build a resized Embedding Module from a provided token Embedding Module. Increasing the size will add newly initialized vectors at th

HP search setup code

HP search setup code

Returns tokenized answer spans that better match the annotated answer.

This function sets a new seed for the attention layer to make dropout deterministic for both forward calls: 1 normal forward

This function sets a new seed for the feed forward layer to make dropout deterministic for both forward calls: 1 normal forwa

Identical to the XLM create_sinusoidal_embeddings except features are not interleaved. The cos features are in the 2nd half of the vector. [di

Initialize the weights

Checks whether CP is the codepoint of a CJK character.

Checks whether the last character in text is one of a punctuation, control or whitespace character.

Checks whether the first character in text is one of a punctuation, control or whitespace character.

length and attention head size dim normalization

length normalization

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, embedding

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [..., hidden_size]

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Computes logits by running inputs through a linear layer. Args: inputs: A float32 tensor with shape [batch_size, length, hidden_si

Check if this is the 'max context' doc span for the token.

shift every row 1 step right, converting columns into diagonals. Example: chunked_hidden_states: [ 0.4983, 2.6918, -0.0071, 1.

shift every row 1 step right, converting columns into diagonals. Example: chunked_hidden_states: [ 0.4983, 2.6918, -0.0071, 1.

pads rows and then flips rows and columns

pads rows and then flips rows and columns

A helper function to pad tokens and mask to work with implementation of Longformer selfattention.

A helper function to pad tokens and mask to work with implementation of Longformer self-attention.

Parse arguments and tokenize, adding an EOS token at the end of the user input

Prepare masks that ignore padding tokens in the decoder and a causal mask for the decoder if none are provided. This mimics the default behavior i

Prepare masks that ignore padding tokens in the decoder and a causal mask for the decoder if none are provided. This mimics the default behavior i

Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel

Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transf

Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transf

Remove entities from text by converting them to their corresponding unicode character. :param text: a unicode string or a byte string en