Method TokenToId

data_utils/tokenization.py:767–769

convert sentencpiece token to Id

(self, token)

        return self.sp.IdToPiece(Id)

    def TokenToId(self, token):
        """convert sentencpiece token to Id"""
        return self.sp.PieceToId(token)

    def DecodeIds(self, Ids):
        """converts ids to a text string"""