MCPcopy Create free account
hub / github.com/SooLab/CGFormer / _is_whitespace

Function _is_whitespace

bert/tokenization_utils.py:47–56  ·  view source on GitHub ↗

Checks whether `chars` is a whitespace character.

(char)

Source from the content-addressed store, hash-verified

45
46
47def _is_whitespace(char):
48 """Checks whether `chars` is a whitespace character."""
49 # \t, \n, and \r are technically contorl characters but we treat them
50 # as whitespace since they are generally considered as such.
51 if char == " " or char == "\t" or char == "\n" or char == "\r":
52 return True
53 cat = unicodedata.category(char)
54 if cat == "Zs":
55 return True
56 return False
57
58
59def _is_control(char):

Callers 3

_clean_textMethod · 0.85
_is_end_of_wordFunction · 0.85
_is_start_of_wordFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected