MCPcopy Create free account
hub / github.com/LiteSSLHub/DisCo / BertTokenizer

Class BertTokenizer

src/pytorch_pretrained_bert/tokenization.py:75–155  ·  view source on GitHub ↗

Runs end-to-end tokenization: punctuation splitting + wordpiece

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers

nothing calls this directly

Calls

no outgoing calls

Tested by

no test coverage detected