Method
__init__
(
self,
tokenizer,
ws_start: int = None,
ws_len: int = None,
)
Source from the content-addressed store, hash-verified
| 218 | """Wrapper for Hugging Face tokenizer.""" |
| 219 | |
| 220 | def __init__( |
| 221 | self, |
| 222 | tokenizer, |
| 223 | ws_start: int = None, |
| 224 | ws_len: int = None, |
| 225 | ): |
| 226 | super(HgfTokenizerWrapper, self).__init__(tokenizer.__class__.__name__) |
| 227 | self.tokenizer = tokenizer |
| 228 | self.ws_start = ws_start |
| 229 | self.ws_len = ws_len |
| 230 | |
| 231 | def tokenize(self, text): |
| 232 | if self.ws_start: |
Callers
nothing calls this directly
Tested by
no test coverage detected