MCPcopy Create free account
hub / github.com/tensorflow/lingvo / StringsToIds

Method StringsToIds

lingvo/core/tokenizers.py:60–92  ·  view source on GitHub ↗

Tokenize strs into vocab ids. Args: strs: A vector of strings. max_length: An int providing the max_length for strs. external_append_eos: Bool or None. If None, will be ignored and `params.append_eos` will be used. If bool, will determine if an eos symbol will

(self,
                   strs,
                   max_length,
                   external_append_eos=None,
                   languages=None)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 14

testStringsToIdsMethod · 0.45
testStringsTokenIdsMethod · 0.45
testIdsToStringsMethod · 0.45
ProcMethod · 0.45
BuildInputBatchMethod · 0.45
_ProcessSingleExampleMethod · 0.45
_ProcessLineMethod · 0.45

Calls 1

_StringsToIdsImplMethod · 0.95

Tested by 5

testStringsToIdsMethod · 0.36
testStringsTokenIdsMethod · 0.36
testIdsToStringsMethod · 0.36