MCPcopy Create free account
hub / github.com/tensorflow/lingvo / Encode

Method Encode

lingvo/core/wpm_encoder.py:140–168  ·  view source on GitHub ↗

Converts string `text` to integer ids and the encoded string. Encoding includes prefixing the beginning-of-word token to each word. Returns: (ids, tokens) where ids is the encoded integer ids and tokens is the encoded string.

(self, text)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 7

_RunEncodingFunction · 0.95
testMergeProbMethod · 0.95
_TokenizeOneSentenceMethod · 0.80
testDittoMethod · 0.80
testEmptyMethod · 0.80
testWithBackslashMethod · 0.80
testWithUnicodeMethod · 0.80

Calls 4

_TokenToStringMethod · 0.95
splitMethod · 0.80
sizeMethod · 0.80
while_loopMethod · 0.80

Tested by 5

testMergeProbMethod · 0.76
testDittoMethod · 0.64
testEmptyMethod · 0.64
testWithBackslashMethod · 0.64
testWithUnicodeMethod · 0.64