MCPcopy Create free account
hub / github.com/PaddlePaddle/Research / _resplit

Function _resplit

NLP/Text2SQL-BASELINE/tools/evaluation/utils.py:83–94  ·  view source on GitHub ↗

resplit

(tmp_tokens, fn_split, fn_omit)

Source from the content-addressed store, hash-verified

81 return fields
82
83 def _resplit(tmp_tokens, fn_split, fn_omit):
84 """resplit"""
85 new_tokens = []
86 for token in tmp_tokens:
87 token = token.strip()
88 if fn_omit(token):
89 new_tokens.append(token)
90 elif re.match(r'\d\d\d\d-\d\d(-\d\d)?', token):
91 new_tokens.append('"%s"' % (token))
92 else:
93 new_tokens.extend(fn_split(token))
94 return new_tokens
95
96 def _split_aggs(tmp_tokens):
97 """split aggs in select"""

Callers 1

tokenize_NL2SQLFunction · 0.70

Calls 2

extendMethod · 0.80
appendMethod · 0.45

Tested by

no test coverage detected