MCPcopy Create free account
hub / github.com/google-research/language / __init__

Method __init__

language/realm/featurization.py:415–428  ·  view source on GitHub ↗
(self, vocab_path, do_lower_case)

Source from the content-addressed store, hash-verified

413 """Tokenizes text."""
414
415 def __init__(self, vocab_path, do_lower_case):
416 if isinstance(vocab_path, bytes):
417 vocab_path = vocab_path.decode()
418
419 self.vocab_path = vocab_path
420 self.do_lower_case = do_lower_case
421
422 self._base_tokenizer = tokenization.FullTokenizer(
423 vocab_file=vocab_path, do_lower_case=do_lower_case)
424
425 # Look up special tokens.
426 self.cls_id, self.sep_id, self.mask_id = (
427 self._base_tokenizer.convert_tokens_to_ids(['[CLS]', '[SEP]',
428 '[MASK]']))
429
430 def tokenize(self, text, compute_token_boundaries=True):
431 """Returns a list of Tokens."""

Callers

nothing calls this directly

Calls 2

decodeMethod · 0.45
convert_tokens_to_idsMethod · 0.45

Tested by

no test coverage detected