MCPcopy Create free account
hub / github.com/LargeWorldModel/LWM / tokenizer

Method tokenizer

lwm/data.py:317–318  ·  view source on GitHub ↗
(self)

Source from the content-addressed store, hash-verified

315
316 @property
317 def tokenizer(self):
318 return self._tokenizer
319
320 @property
321 def text_processor(self):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected