MCPcopy Index your code
hub / github.com/ddbourgin/numpy-ml / bytes_to_chars

Function bytes_to_chars

numpy_ml/preprocessing/nlp.py:141–148  ·  view source on GitHub ↗

Decode bytes (represented as an integer between 0 and 255) to characters in the specified encoding.

(byte_list, encoding="utf-8")

Source from the content-addressed store, hash-verified

139
140
141def bytes_to_chars(byte_list, encoding="utf-8"):
142 """
143 Decode bytes (represented as an integer between 0 and 255) to characters in
144 the specified encoding.
145 """
146 hex_array = [hex(a).replace("0x", "") for a in byte_list]
147 hex_array = " ".join([h if len(h) > 1 else f"0{h}" for h in hex_array])
148 return bytearray.fromhex(hex_array).decode(encoding)
149
150
151def tokenize_chars(line, lowercase=True, filter_punctuation=True, **kwargs):

Callers 1

inverse_transformMethod · 0.85

Calls 1

decodeMethod · 0.80

Tested by

no test coverage detected