MCPcopy Create free account
hub / github.com/YunwenTechnology/QueryGeneration / convert_tokens_to_ids

Function convert_tokens_to_ids

tokenization/tokenization.py:131–132  ·  view source on GitHub ↗
(vocab, tokens)

Source from the content-addressed store, hash-verified

source not stored for this graph (policy: none)

Callers

nothing calls this directly

Calls 1

convert_by_vocabFunction · 0.85

Tested by

no test coverage detected