MCPcopy Create free account
hub / github.com/dateutil/dateutil / get_token

Method get_token

src/dateutil/parser/_parser.py:73–180  ·  view source on GitHub ↗

This function breaks the time string into lexical units (tokens), which can be parsed by the parser. Lexical units are demarcated by changes in the character set, so any continuous string of letters is considered one unit, any continuous string of numbers is consider

(self)

Source from the content-addressed store, hash-verified

71 self.eof = False
72
73 def get_token(self):
74 """
75 This function breaks the time string into lexical units (tokens), which
76 can be parsed by the parser. Lexical units are demarcated by changes in
77 the character set, so any continuous string of letters is considered
78 one unit, any continuous string of numbers is considered one unit.
79
80 The main complication arises from the fact that dots ('.') can be used
81 both as separators (e.g. "Sep.20.2009") or decimal points (e.g.
82 "4:30:21.447"). As such, it is necessary to read the full context of
83 any dot-separated strings before breaking it into tokens; as such, this
84 function maintains a "token stack", for when the ambiguous context
85 demands that multiple tokens be parsed at once.
86 """
87 if self.tokenstack:
88 return self.tokenstack.pop(0)
89
90 seenletters = False
91 token = None
92 state = None
93
94 while not self.eof:
95 # We only realize that we've reached the end of a token when we
96 # find a character that's not part of the current token - since
97 # that character may be part of the next token, it's stored in the
98 # charstack.
99 if self.charstack:
100 nextchar = self.charstack.pop(0)
101 else:
102 nextchar = self.instream.read(1)
103 while nextchar == '\x00':
104 nextchar = self.instream.read(1)
105
106 if not nextchar:
107 self.eof = True
108 break
109 elif not state:
110 # First character of the token - determines if we're starting
111 # to parse a word, a number or something else.
112 token = nextchar
113 if self.isword(nextchar):
114 state = 'a'
115 elif self.isnum(nextchar):
116 state = '0'
117 elif self.isspace(nextchar):
118 token = ' '
119 break # emit token
120 else:
121 break # emit token
122 elif state == 'a':
123 # If we've already started reading a word, we keep reading
124 # letters until we find something that's not part of a word.
125 seenletters = True
126 if self.isword(nextchar):
127 token += nextchar
128 elif nextchar == '.':
129 token += nextchar
130 state = 'a.'

Callers 1

__next__Method · 0.95

Calls 8

iswordMethod · 0.95
isnumMethod · 0.95
isspaceMethod · 0.95
appendMethod · 0.80
countMethod · 0.80
splitMethod · 0.80
replaceMethod · 0.80
readMethod · 0.45

Tested by

no test coverage detected