Isolate a glossary present inside a word. Returns a list of subwords. In which all 'glossary' glossaries are isolated For example, if

Return the list of available models

Read in multiple models for ensemble.

display statistics to tensorboard

Learn num_symbols BPE operations from vocabulary, and write to outfile.

Returns the learning decay function from options.

Convert alignment to match detokenized src/tgt (or not). Args: src (str): The tokenized source sequence. tgt (str): T

De-tokenize the sequence (or not) Same args/returns as :func:`tokenize()`

De-tokenize (or not) the sequence (with alignment). Args: sequence (str): The sequence to detokenize, possible with

Postprocess the sequence (or not)

Preprocess the sequence (or not)

Tokenize the sequence (or not). Same args/returns as `tokenize`

We complete and reorder the list of forbidden_tokens

Turn a batch of examples that use this field into a Variable. If the field has ``include_lengths=True``, a tensor of lengths will be

Turn a batch of examples that use this field into a Variable. If the field has ``include_lengths=True``, a tensor of lengths will be i

Write out statistics to stdout. Args: step (int): current step n_batch (int): total batches start (int): sta

Parse the option set passed by the user using `onmt.opts` Args: opt (dict): Options passed by the user Returns:

Preprocess a single sequence. Args: sequence (str): The sequence to process. Returns: sequence (str): The po

Preprocess a single sequence. Args: sequence (str): The sequence to preprocess. Returns: sequence (str): The

Preprocess data. Args: x (str): A sentence string (words joined by whitespace). Returns: List[List[str]]: A

Convert outputs of preprocess into Tensors. Args: batch (List[List[List[str]]]): A list of length batch size. Eac

Reads the provided files in batches

Yield images and their frame number from a video file.

read vocabulary file produced by get_vocab.py, and filter according to frequency threshold.

Rebuild proper segment packages based on initial n_seg.

Replace all occurrences of a symbol pair ('A', 'B') with a new symbol 'AB

Report stats of a step Args: train_stats(Statistics): training stats valid_stats(Statistics): validation sta

This is the user-defined batch-level traing progress report function. Args: step(int): current step count.

Select next tokens randomly from the top k possible next tokens. Samples from a categorical distribution over the ``keep_topk`` words using t

Main entry point for model saver It wraps the `_save` method with checks and apply `keep_checkpoint` related logic

Args: h_t (FloatTensor): sequence of queries ``(batch, tgt_len, dim)`` h_s (FloatTensor): sequence of sources ``(batch, s

segment single sentence (whitespace-tokenized string) with BPE encoding

linear transform for 3-d tensor

Builds a single iterator yielding every shard of every corpus.

Args: state: A dictionary which corresponds to the output of *LossCompute._make_shard_state(). The values for

Calculate ROUGE scores of sequences passed as an iterator e.g. a list of str, an open file, StringIO or even sys.stdin

Convert subword alignment to word alignment. Args: src (string): tokenized sentence in source language. tgt (string): tokenized s

Translate content of ``src`` and get gold scores from ``tgt``. Args: src: See :func:`self.src_reader.read()`. tgt: Se

Minimally update the indices and frequency of symbol pairs if we merge a pair of symbols, only pairs that overlap with occurrences of this pa

Validate model. valid_iter: validate data iterator Returns: :obj:`nmt.Statistics`: validation loss statistics

Return the length of a video.

Compute the forward loss, possibly in shards in which case this method also runs the backward pass and returns ``None`` as the loss va

Update the internal state of early stopping mechanism, whether to continue training or stop the train procedure. Che