MCPcopy Create free account
hub / github.com/YunwenTechnology/QueryGeneration / WordpieceTokenizer

Class WordpieceTokenizer

tokenization/tokenization.py:287–346  ·  view source on GitHub ↗

Runs WordPiece tokenziation.

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers 1

__init__Method · 0.85

Calls

no outgoing calls

Tested by

no test coverage detected