MCPcopy Create free account
hub / github.com/google-research/language / SQLTokenizer

Class SQLTokenizer

language/xsp/data_preprocessing/sql_utils.py:31–41  ·  view source on GitHub ↗

A SQL tokenizer.

Source from the content-addressed store, hash-verified

29
30
31class SQLTokenizer(object):
32 """A SQL tokenizer."""
33
34 def __init__(self):
35 pass
36
37 def tokenize(self, sql):
38 """Tokenizes a SQL query into a list of SQL tokens."""
39 return [str(token).strip() for token in \
40 sqlparse.sql.TokenList(sqlparse.parse(sql)[0].tokens).flatten() \
41 if str(token).strip()]
42
43
44def anonymize_aliases(sql):

Callers 1

anonymize_aliasesFunction · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…