MCPcopy Create free account
hub / github.com/tensorflow/models / convert_to_unicode

Function convert_to_unicode

official/nlp/tools/tokenization.py:84–101  ·  view source on GitHub ↗

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

(text)

Source from the content-addressed store, hash-verified

82
83
84def convert_to_unicode(text):
85 """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
86 if six.PY3:
87 if isinstance(text, str):
88 return text
89 elif isinstance(text, bytes):
90 return text.decode("utf-8", "ignore")
91 else:
92 raise ValueError("Unsupported string type: %s" % (type(text)))
93 elif six.PY2:
94 if isinstance(text, str):
95 return text.decode("utf-8", "ignore")
96 elif isinstance(text, unicode):
97 return text
98 else:
99 raise ValueError("Unsupported string type: %s" % (type(text)))
100 else:
101 raise ValueError("Not running on Python2 or Python 3?")
102
103
104def printable_text(text):

Callers 3

load_vocabFunction · 0.85
tokenizeMethod · 0.85
tokenizeMethod · 0.85

Calls 1

decodeMethod · 0.45

Tested by

no test coverage detected