MCPcopy Create free account
hub / github.com/google-research/language / tokenizer

Method tokenizer

language/serene/preprocessing.py:292–293  ·  view source on GitHub ↗
(self)

Source from the content-addressed store, hash-verified

290
291 @property
292 def tokenizer(self):
293 return self._tokenizer
294
295 def _oov_bucket(self, token):
296 if self._oov_buckets <= 0:

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected