MCPcopy Create free account
hub / github.com/google-research/language / id2text

Function id2text

language/labs/exemplar_decoding/utils/data.py:92–100  ·  view source on GitHub ↗
(seq, vocab, use_bpe=False)

Source from the content-addressed store, hash-verified

90
91
92def id2text(seq, vocab, use_bpe=False):
93 tokens = [vocab.id2word(i) for i in seq]
94 tokens = [x for x in tokens
95 if x not in [STOP_DECODING, SEPARATOR, SENTENCE_START,
96 PAD_TOKEN, START_DECODING]]
97 text = " ".join(tokens)
98 if use_bpe:
99 text = recover_tokenization(text)
100 return remove_repetive_unigram(text)
101
102
103def input_function(is_train, vocab, hps):

Callers 3

rouge_lFunction · 0.90
rouge_nFunction · 0.90
_do_printFunction · 0.90

Calls 3

recover_tokenizationFunction · 0.85
remove_repetive_unigramFunction · 0.85
id2wordMethod · 0.80

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…