Add an item to be preprocessed.

Select ``target`` of size ``(batch_size, sequence_length, embedding_size)`` with indices of size ``(batch_size, d)``. The mask is optional of siz

Select ``target`` of size ``(batch_size, sequence_length, embedding_size)`` with ``indices`` of size ``(batch_size, d_1, ***, d_n)``. Args: t

Select ``target`` of size ``(batch_size, sequence_length, d2, d3, ... ,dn)`` with indices of size ``(batch, index_of_sequence)``. Args: target

Find the valid FROM clasue with beam search

Clear the preprocessed items

model: a nl2code decoder

Returns a torch.data.utils.Dataset instance.

Same default as PyTorch

Compute the entropy of a probability distribution

Args: x (Tensor): input to the layer of shape `(seq_len, batch, embed_dim)` encoder_padding_mask (ByteTensor): binary

Args: input_ids (LongTensor): tokens in the source language of shape `(batch, src_len)` attention_mas

Includes several features from "Jointly Learning to Align and Translate with Transformer Models" (Garg et al., EMNLP 2019).

Input shape: Time(SeqLen) x Batch x Channel

Input is expected to be of size [bsz x seqlen].

r""" masked_lm_labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`, defaults to :obj:`None`):

r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`, defaults to :obj:`None`): Labels for computing the