:param ref - a list of tokens of the reference :param hyps - a list of tokens of the hypothesis :return maxbleu - recall ble

:param pred - ndarray [batch_size x seqlen] :param pred_lens - list of integers :param ref - ndarray [batch_size x seqlen]

:param pred - ndarray [batch_size x seqlen] :param pred_lens - list of integers :param ref - ndarray [batch_size x seqlen]

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (vocabulary

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (voc

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (voc

Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (vocabulary

:param word2vec - a numpy array of word2vec with shape [vocab_size x emb_size]

:param word2vec - a numpy array of word2vec with shape [vocab_size x emb_size]

:param config: :param add_input: :param add_attn: :param attn_proj_vary: :param cond: whether add label embe

Code Book for VQ-VAE :param num_embeddings: :param embedding_dim: :param commitment_cost:

Constructs a InputExample. Args: guid: Unique id for the example. text_a: string. The untokenized text of the first s

Creates a batch out of samples for direct input

Loads a data file into a list of `InputBatch`s `cls_token_at_end` define the location of the CLS token: - False (Default, BERT/XL