MCPcopy Create free account
hub / github.com/SooLab/CGFormer / bos_token

Method bos_token

bert/tokenization_utils_base.py:718–723  ·  view source on GitHub ↗

Beginning of sentence token (string). Log an error if used while not having been set.

(self)

Source from the content-addressed store, hash-verified

716
717 @property
718 def bos_token(self):
719 """ Beginning of sentence token (string). Log an error if used while not having been set. """
720 if self._bos_token is None and self.verbose:
721 logger.error("Using bos_token, but it is not set yet.")
722 return None
723 return str(self._bos_token)
724
725 @property
726 def eos_token(self):

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected