MCPcopy Create free account
hub / github.com/apache/singa / convert_to_unicode

Function convert_to_unicode

examples/onnx/bert/tokenization.py:77–92  ·  view source on GitHub ↗

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

(text)

Source from the content-addressed store, hash-verified

75
76
77def convert_to_unicode(text):
78 """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
79 if six.PY3:
80 if isinstance(text, str):
81 return text
82 elif isinstance(text, bytes):
83 return text.decode("utf-8", "ignore")
84 else:
85 raise ValueError("Unsupported string type: %s" % (type(text)))
86 elif six.PY2:
87 if isinstance(text, str):
88 return text.decode("utf-8", "ignore")
89 else:
90 raise ValueError("Unsupported string type: %s" % (type(text)))
91 else:
92 raise ValueError("Not running on Python2 or Python 3?")
93
94
95def printable_text(text):

Callers 2

tokenizeMethod · 0.85
tokenizeMethod · 0.85

Calls 1

typeFunction · 0.85

Tested by

no test coverage detected