MCPcopy Create free account
hub / github.com/amazon-science/ReFinED / tokenize

Method tokenize

src/refined/doc_preprocessing/preprocessor.py:106–107  ·  view source on GitHub ↗
(self, text: str)

Source from the content-addressed store, hash-verified

104
105 @abstractmethod
106 def tokenize(self, text: str):
107 pass
108
109 @abstractmethod
110 def get_transformer_model(self):

Callers 3

tokenize_and_preserve_labelsFunction · 0.45
from_textMethod · 0.45
from_text_with_spansMethod · 0.45

Calls

no outgoing calls

Tested by

no test coverage detected