MCPcopy Create free account
hub / github.com/aboutcode-org/scancode-toolkit / toascii

Function toascii

src/commoncode/text.py:92–118  ·  view source on GitHub ↗

Convert a Unicode or byte string to ASCII characters, including replacing accented characters with their non-accented equivalent. If `translit` is False use the Unicode NFKD equivalence. If `translit` is True, use a transliteration with the unidecode library. Non ISO-Latin and

(s, translit=False)

Source from the content-addressed store, hash-verified

90
91
92def toascii(s, translit=False):
93 """
94 Convert a Unicode or byte string to ASCII characters, including replacing
95 accented characters with their non-accented equivalent.
96
97 If `translit` is False use the Unicode NFKD equivalence.
98 If `translit` is True, use a transliteration with the unidecode library.
99
100 Non ISO-Latin and non ASCII characters are stripped from the output. When no
101 transliteration is possible, the resulting character is replaced by an
102 underscore "_".
103
104 For Unicode NFKD equivalence, see http://en.wikipedia.org/wiki/Unicode_equivalence
105 The convertion may NOT preserve the original string length and with NFKD some
106 characters may be deleted.
107 Inspired from: http://code.activestate.com/recipes/251871/#c10 by Aaron Bentley.
108 """
109 if not isinstance(s, str):
110 s = as_unicode(s)
111 if translit:
112 converted = unidecode(s)
113 else:
114 converted = unicodedata.normalize("NFKD", s)
115
116 converted = converted.replace("[?]", "_")
117 converted = converted.encode("ascii", "ignore")
118 return converted.decode("ascii")
119
120
121def python_safe_name(s):

Callers 7

transliterateMethod · 0.90
get_generated_code_hintFunction · 0.90
portable_filenameFunction · 0.90
path_progress_messageFunction · 0.90
findFunction · 0.90
prepare_text_lineFunction · 0.90
python_safe_nameFunction · 0.85

Calls 5

as_unicodeFunction · 0.70
normalizeMethod · 0.45
replaceMethod · 0.45
encodeMethod · 0.45
decodeMethod · 0.45

Tested by

no test coverage detected