Function tokenize_docstring

src/codetext/parser/language_parser.py:20–21 · view source on GitHub ↗

(docstring: str)

Source from the content-addressed store, hash-verified

18
19
20	def tokenize_docstring(docstring: str) -> List[str]:
21	return [t for t in DOCSTRING_REGEX_TOKENIZER.findall(str(docstring)) if t is not None and len(t) > 0]
22
23
24	def tokenize_code(node, blob: str, nodes_to_exclude: Optional[Set]=None) -> List:

check_contain_many_special_charFunction · 0.85

check_contain_little_unique_wordsFunction · 0.85

split_all_sepcial_caseFunction · 0.85

no outgoing calls

no test coverage detected