MCPcopy Create free account
hub / github.com/Tiiny-AI/PowerInfer / test-tokenizer-0-falcon.py

File test-tokenizer-0-falcon.py

tests/test-tokenizer-0-falcon.py:None–None  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

1# tests with BPE tokenizer
2
3import os
4import sys

Callers

nothing calls this directly

Calls 6

strFunction · 0.85
joinMethod · 0.80
printFunction · 0.50
encodeMethod · 0.45
decodeMethod · 0.45
writeMethod · 0.45

Tested by

no test coverage detected