MCPcopy Create free account
hub / github.com/TIGER-AI-Lab/LongRAG / tokenize

Method tokenize

utils/tokenizers.py:134–135  ·  view source on GitHub ↗
(self, text)

Source from the content-addressed store, hash-verified

132 """
133
134 def tokenize(self, text):
135 raise NotImplementedError
136
137 def shutdown(self):
138 pass

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected