Save a resumable checkpoint. Args: step (int): step number Returns: (object, str): * checkpoint

Yield a `list` containing `shard_size` line of `path`.

Simple function to start report manager (if any)

Translate a batch of sentences step by step using cache. Args: batch: a batch of sentences, yield by data iterator. s

Gathers arbitrary data from all nodes into a list.

Gather a `Statistics` list accross all processes/nodes Args: stat_list(list([`Statistics`])): list of statistics objects

Collate frames into batches of equal length.

Various decoder dispatcher function. Args: opt: the option in current environment. embeddings (Embeddings): vocab embeddings

Various encoder dispatcher function. Args: opt: the option in current environment. embeddings (Embeddings): vocab embeddings

Builds the PyTorch optimizer. We use the default parameters for Adam that are suggested by the original paper https://arxiv.org/pdf/1412.6980

Simplify `Trainer` creation based on user `opt`s* Args: opt (:obj:`Namespace`): user options (usually from argument parsing)

Check if there are existing .pt files to avoid overwriting them

Check for each segment in word if it is in-vocabulary, and segment OOV segments into smaller units by reversing the BPE merge operations

Clone a model `model_id`. Different options may be passed. If `opt` is None, it will use the same set of options

Returns the correct ContextGate class

Computes the cumulative average as described in :cite:`DBLP:journals/corr/abs-1805-00631` -- Equations (1) (5) (6) Args:

Builds the mask to compute the cumulative average as described in :cite:`DBLP:journals/corr/abs-1805-00631` -- Figure 3 Args

Get default arguments added to a parser by all ``*args``.

Detokenize a single sequence Same args/returns as :func:`tokenize()`

Encode word based on list of BPE merge operations, which are applied consecutively

Extract a batched align_matrix into its src indice alignment lists, with tgt_mask to filter out invalid tgt position as EOS/PAD. BOS alre

Returns the list of function arguments name.

Generate the clipped relative positions matrix for a given length and maximum relative positions

Args: src_data_type: type of the source input. Options are [text|img|audio]. n_src_feats (int): the number of source features (no

Count frequency of all symbol pairs, and create index

utility for retrieving polyak averaged params Update average

Read text and return dictionary that encodes vocabulary

Isolate a glossary present inside a word. Returns a list of subwords. In which all 'glossary' glossaries are isolated For example, if

Return the list of available models

Load a model given a set of options

Read in multiple models for ensemble.

display statistics to tensorboard

Learn num_symbols BPE operations from vocabulary, and write to outfile.

Returns the learning decay function from options.

Convert alignment to match detokenized src/tgt (or not). Args: src (str): The tokenized source sequence. tgt (str): T

De-tokenize the sequence (or not) Same args/returns as :func:`tokenize()`

De-tokenize (or not) the sequence (with alignment). Args: sequence (str): The sequence to detokenize, possible with

Postprocess the sequence (or not)

Preprocess the sequence (or not)

Tokenize the sequence (or not). Same args/returns as `tokenize`

Write out statistics to stdout. Args: step (int): current step n_batch (int): total batches start (int): sta

Parse the option set passed by the user using `onmt.opts` Args: opt (dict): Options passed by the user Returns:

Preprocess a single sequence. Args: sequence (str): The sequence to process. Returns: sequence (str): The po

Preprocess a single sequence. Args: sequence (str): The sequence to preprocess. Returns: sequence (str): The

Reads the provided files in batches

Yield images and their frame number from a video file.

read vocabulary file produced by get_vocab.py, and filter according to frequency threshold.

Replace all occurrences of a symbol pair ('A', 'B') with a new symbol 'AB

Report stats of a step Args: train_stats(Statistics): training stats valid_stats(Statistics): validation sta

This is the user-defined batch-level traing progress report function. Args: step(int): current step count.

Select next tokens randomly from the top k possible next tokens. Samples from a categorical distribution over the ``keep_topk`` words using t

Main entry point for model saver It wraps the `_save` method with checks and apply `keep_checkpoint` related logic

Args: h_t (FloatTensor): sequence of queries ``(batch, tgt_len, dim)`` h_s (FloatTensor): sequence of sources ``(batch, s

segment single sentence (whitespace-tokenized string) with BPE encoding

linear transform for 3-d tensor

Builds a single iterator yielding every shard of every corpus.