Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/google-research/language
/ tokenizer
Method
tokenizer
language/serene/preprocessing.py:292–293 ·
view source on GitHub ↗
(self)
Source
from the content-addressed store, hash-verified
290
291
@property
292
def
tokenizer(self):
293
return
self._tokenizer
294
295
def
_oov_bucket(self, token):
296
if
self._oov_buckets <= 0:
Callers
nothing calls this directly
Calls
no outgoing calls
Tested by
no test coverage detected