output1 - True - algorithm for matrix, False - vector; output2 - need reshape

Compute loss between 2 partial alignment matrix.

Create copy-vocab and numericalize with it. In-place adds ``"src_map"`` to ``example``. That is the copy-vocab numericalization of the tokeni

Make shard state dictionary for shards() to return iterable shards for efficient loss computation. Subclass must define this

Gather statistics in multi-processes cases Args: stat(:obj:onmt.utils.Statistics): a Statistics object to gather

Simple function to report training stats (if report_manager is set) see `onmt.utils.ReportManagerBase.report_training` for doc

Loads a vocabulary from the given path. Args: vocab_path (str): Path to utf-8 text file containing vocabulary. Each token sho

Simple function to report stats (if report_manager is set) see `onmt.utils.ReportManagerBase.report_step` for doc

To be overridden

Remove a checkpoint Args: name(str): name that indentifies the checkpoint (it may be a filepath)

Private helper for running the specific RNN forward pass. Must be overriden by all subclasses. Args: tgt (LongTe

Save a resumable checkpoint. Args: step (int): step number Returns: (object, str): * checkpoint

Yield a `list` containing `shard_size` line of `path`.

Simple function to start report manager (if any)

Translate a batch of sentences step by step using cache. Args: batch: a batch of sentences, yield by data iterator. s

Gathers arbitrary data from all nodes into a list.

Gather a `Statistics` list accross all processes/nodes Args: stat_list(list([`Statistics`])): list of statistics objects

Collate frames into batches of equal length.

In place add noise related fields i.e.: - word_start - end_of_sentence

Builds the PyTorch optimizer. We use the default parameters for Adam that are suggested by the original paper https://arxiv.org/pdf/1412.6980

Simplify `Trainer` creation based on user `opt`s* Args: opt (:obj:`Namespace`): user options (usually from argument parsing)

Check if there are existing .pt files to avoid overwriting them

Check for each segment in word if it is in-vocabulary, and segment OOV segments into smaller units by reversing the BPE merge operations

Clone a model `model_id`. Different options may be passed. If `opt` is None, it will use the same set of options

Returns the correct ContextGate class

Computes the cumulative average as described in :cite:`DBLP:journals/corr/abs-1805-00631` -- Equations (1) (5) (6) Args:

Builds the mask to compute the cumulative average as described in :cite:`DBLP:journals/corr/abs-1805-00631` -- Figure 3 Args

Get default arguments added to a parser by all ``*args``.

Encode word based on list of BPE merge operations, which are applied consecutively

Extract a batched align_matrix into its src indice alignment lists, with tgt_mask to filter out invalid tgt position as EOS/PAD. BOS alre

Returns the list of function arguments name.

Generate the clipped relative positions matrix for a given length and maximum relative positions

Creates a CTranslate2 model specification from the model options.

Args: src_data_type: type of the source input. Options are [text|img|audio]. n_src_feats (int): the number of source features (no

Count frequency of all symbol pairs, and create index

utility for retrieving polyak averaged params Update average

Read text and return dictionary that encodes vocabulary