Splits punctuation on a piece of text.

Returns: A ``tf.data.Dataset`` containing the task-specific features.

Adds whitespace around any CJK character.

Loads a pre-existing dictionary from a text file and adds its symbols to this instance.

Loads a pre-existing dictionary from a text file and adds its symbols to this instance.

Implement in subclasses of :class:`~transfomers.PreTrainedModel` for custom behavior to adjust the logits in the generate method.

Build shared word embedding layer

Build shared word embedding layer

Build shared word embedding layer

Build shared word embedding layer

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens

A map of modules from TF to PyTorch. This time I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.

A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.

Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code bar

cache hidden states into memory.

Copied from fairseq for no_repeat_ngram in beam_search

Uses sacrebleu's corpus_bleu implementation.

1's in the lower triangle, counting from the lower right corner. Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't produc

1's in the lower triangle, counting from the lower right corner. Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't produc

Check onnxruntime is installed and if the installed version match is recent enough. Raises: ImportError: If onnxruntime is not instal

Compute binned relative position bias

Compute binned relative position bias

Convert the pipeline object to the ONNX Intermediate Representation (IR) format. Args: framework: The framework the pipeline is backe

Copy/paste/tweak model's weights to our BERT structure.

Relies on the assumption that there are no language codes like pt_br in models that are not in GROUP_TO_OPUS_NAME.

Export a PyTorch backed pipeline to ONNX Intermediate Representation (IR) Args: nlp: The pipeline to be exported opset: The a

:param model:BertModel Pytorch model instance to be converted :param ckpt_dir: Tensorflow model directory :param model_name: model name

Copy/paste/tweak roberta's weights to our BERT structure.

Export a TensorFlow backed pipeline to ONNX Intermediate Representation (IR) Args: nlp: The pipeline to be exported opset: Th