Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/google-research/language
/ tokens
Method
tokens
language/serene/preprocessing.py:280–281 ·
view source on GitHub ↗
(self)
Source
from the content-addressed store, hash-verified
278
279
@property
280
def
tokens(self):
281
return
list(self._vocab_list)
282
283
@property
284
def
oov_token(self):
Callers
nothing calls this directly
Calls
no outgoing calls
Tested by
no test coverage detected