Functions — github.com/M4THYOU/TokenDagger

CodeHub MCPcopy Create free account

hub / github.com/M4THYOU/TokenDagger / functions

Functions724 in github.com/M4THYOU/TokenDagger

⨍ Functions 724◇ Types & classes 198

↓ 1 callersMethodload_llama_config

Load Llama 4 configuration from the codebase.

tests/code_performance_benchmark.py:156

↓ 1 callersMethodload_llama_config

Load Llama 4 configuration from the codebase.

tests/performance_benchmark.py:80

↓ 1 callersMethodload_mistral_config

Load Mistral's Tekken 7 configuration from the codebase. https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/tree/main

tests/throughput_test.py:106

↓ 1 callersMethodload_mistral_config

Load Mistral's Tekken 7 configuration from the codebase. Odd notes about Mistral and Tekken configs. tekken.json inc

tests/test_tokendagger_vs_tiktoken.py:61

↓ 1 callersMethodload_mistral_config

Load Mistral's Tekken 7 configuration from the codebase. Odd notes about Mistral and Tekken configs. tekken.json inc

tests/code_performance_benchmark.py:169

↓ 1 callersMethodload_mistral_config

Load Mistral's Tekken 7 configuration from the codebase. Odd notes about Mistral and Tekken configs. tekken.json inc

tests/performance_benchmark.py:93

↓ 1 callersMethodload_special_tokens

Load special tokens from tokenizer_config.json.

tests/throughput_test.py:170

↓ 1 callersMethodload_special_tokens

Load special tokens from tokenizer_config.json.

tests/test_tokendagger_vs_tiktoken.py:141

↓ 1 callersMethodload_special_tokens

Load special tokens from tokenizer_config.json.

tests/code_performance_benchmark.py:249

↓ 1 callersMethodload_special_tokens

Load special tokens from tokenizer_config.json.

tests/performance_benchmark.py:173

↓ 1 callersMethodload_test_texts

Load test texts from input directory.

tests/test_tokendagger_vs_tiktoken.py:210

↓ 1 callersFunctionmain

Main build function.

build_tokendagger.py:26

↓ 1 callersFunctionmain

Main benchmark runner.

tests/throughput_test.py:791

↓ 1 callersFunctionmain

Main test runner.

tests/test_tokendagger_vs_tiktoken.py:490

↓ 1 callersFunctionmain

Main benchmark runner.

tests/code_performance_benchmark.py:593

↓ 1 callersFunctionmain

Main benchmark runner.

tests/performance_benchmark.py:628

↓ 1 callersFunctionoperator+

! @brief add to iterator @pre The iterator is initialized; i.e. `m_object != nullptr`. */

src/nlohmann/json.hpp:13490

↓ 1 callersMethodoperator+

add to iterator

src/nlohmann/json.hpp:13715

↓ 1 callersFunctionoperator+=

@brief add an object to an array @sa https://json.nlohmann.me/api/basic_json/operator+=/

src/nlohmann/json.hpp:22415

↓ 1 callersFunctionoperator-

! @brief subtract from iterator @pre The iterator is initialized; i.e. `m_object != nullptr`. */

src/nlohmann/json.hpp:13512

↓ 1 callersFunctionoperator<

src/nlohmann/json.hpp:2902

↓ 1 callersMethodoperator==

src/nlohmann/json.hpp:12773

↓ 1 callersFunctionparse

src/nlohmann/json.hpp:23344

↓ 1 callersMethodprint_summary

Print test summary.

tests/test_tokendagger_vs_tiktoken.py:428

↓ 1 callersMethodprint_summary_report

Print comprehensive throughput analysis.

tests/throughput_test.py:511

↓ 1 callersMethodprint_summary_report

Print comprehensive analysis of code tokenization performance.

tests/code_performance_benchmark.py:490

↓ 1 callersMethodprint_summary_report

Print comprehensive performance analysis.

tests/performance_benchmark.py:529

↓ 1 callersMethodread_file_safely

Safely read a file with fallback encodings.

tests/code_performance_benchmark.py:321

↓ 1 callersFunctionremove_sign

* Overload to make the compiler happy while it is instantiating * dump_integer for number_unsigned_t. * Must never be called. */

src/nlohmann/json.hpp:18948

↓ 1 callersMethodrun_all_tests

Run all test suites.

tests/test_tokendagger_vs_tiktoken.py:466

↓ 1 callersMethodrun_benchmarks

Run benchmark suite on code files.

tests/code_performance_benchmark.py:428

↓ 1 callersMethodrun_benchmarks

Run comprehensive benchmark suite.

tests/performance_benchmark.py:492

↓ 1 callersMethodrun_decoding_tests

Run decoding comparison tests.

tests/test_tokendagger_vs_tiktoken.py:339

↓ 1 callersMethodrun_encoding_tests

Run encoding comparison tests.

tests/test_tokendagger_vs_tiktoken.py:308

↓ 1 callersMethodrun_full_benchmark

Run the complete throughput benchmark suite.

tests/throughput_test.py:772

↓ 1 callersMethodrun_full_benchmark

Run the complete code benchmark suite.

tests/code_performance_benchmark.py:579

↓ 1 callersMethodrun_full_benchmark

Run the complete benchmark suite.

tests/performance_benchmark.py:614

↓ 1 callersMethodrun_roundtrip_tests

Test encode->decode roundtrip consistency.

tests/test_tokendagger_vs_tiktoken.py:389

↓ 1 callersMethodrun_throughput_benchmarks

Run throughput benchmarks across all thread counts.

tests/throughput_test.py:451

↓ 1 callersFunctionsax_parse

src/nlohmann/json.hpp:23384

↓ 1 callersMethodsetup_tokenizers

Initialize both tokenizers with identical configuration.

tests/throughput_test.py:188

↓ 1 callersMethodsetup_tokenizers

Initialize both tokenizers.

tests/test_tokendagger_vs_tiktoken.py:159

↓ 1 callersMethodsetup_tokenizers

Initialize both tokenizers.

tests/code_performance_benchmark.py:110

↓ 1 callersMethodsetup_tokenizers

Initialize both tokenizers with identical configuration.

tests/performance_benchmark.py:191

↓ 1 callersFunctionskip_bom

! @brief skip the UTF-8 byte order mark @return true iff there is no BOM or the correct BOM has been skipped */

src/nlohmann/json.hpp:8858

↓ 1 callersFunctionsplit

! @brief split the string input to reference tokens @note This function is only called by the json_pointer constructor. All excepti

src/nlohmann/json.hpp:14471

↓ 1 callersFunctionstring

src/nlohmann/json.hpp:6854

↓ 1 callersFunctionto_cbor

@brief create a CBOR serialization of a given JSON value @sa https://json.nlohmann.me/api/basic_json/to_cbor/

src/nlohmann/json.hpp:23548

↓ 1 callersFunctionto_json_tuple_impl

src/nlohmann/json.hpp:5780

↓ 1 callersFunctiontype

@brief return the type of the JSON value (explicit) @sa https://json.nlohmann.me/api/basic_json/type/

src/nlohmann/json.hpp:20596

↓ 1 callersFunctionunescape

src/nlohmann/json.hpp:3005

↓ 1 callersFunctionvalue

! @brief return the value of an iterator @pre The iterator is initialized; i.e. `m_object != nullptr`. */

src/nlohmann/json.hpp:13606

↓ 1 callersMethodwhat

returns the explanatory string

src/nlohmann/json.hpp:4375

FunctionEncoding

TikToken-compatible factory function.

tokendagger/wrapper.py:382

src/tiktoken/hash_table8.hpp:248

src/tiktoken/hash_set8.hpp:232

FunctionJSON_CATCH

src/nlohmann/json.hpp:20000

FunctionJSON_INTERNAL_CATCH

src/nlohmann/json.hpp:21639

FunctionPYBIND11_MODULE

src/py_binding.cpp:7

MethodTiktokenError

src/tiktoken/tiktoken.hpp:34

Initialize the tokenizer. Args: name: Name of the tokenizer pattern: Regex pattern for text splitting

tokendagger/wrapper.py:34

(self, tokenizer_type: str = "llama", thread_counts: List[int] = [1, 2, 4,

tests/throughput_test.py:67

(self, tokenizer_type: str = "llama")

tests/test_tokendagger_vs_tiktoken.py:38

(self, warmup_runs: int = 3, benchmark_runs: int = 25, tokenizer_type: str = "llama")

tests/code_performance_benchmark.py:67

(self, warmup_runs: int = 5, benchmark_runs: int = 100, tokenizer_type: str = "llama")

tests/performance_benchmark.py:68

tokendagger/wrapper.py:138

src/nlohmann/json.hpp:23357

Methodadd_value_labels

tests/throughput_test.py:709

Methodalloc_bucket

src/tiktoken/hash_table8.hpp:1099

Methodalloc_bucket

src/tiktoken/hash_set8.hpp:843

Methodalloc_index

src/tiktoken/hash_table8.hpp:1109

Methodalloc_index

src/tiktoken/hash_set8.hpp:853

@brief access the last element @sa https://json.nlohmann.me/api/basic_json/back/

src/nlohmann/json.hpp:21710

Functionbasic_json array

@brief explicitly create an array from an initializer list @sa https://json.nlohmann.me/api/basic_json/array/

src/nlohmann/json.hpp:20306

Functionbasic_json binary

@brief explicitly create a binary array (without subtype) @sa https://json.nlohmann.me/api/basic_json/binary/

src/nlohmann/json.hpp:20262

Functionbasic_json diff

@brief creates a diff as a JSON patch @sa https://json.nlohmann.me/api/basic_json/diff/

src/nlohmann/json.hpp:24281

Functionbasic_json from_bjdata

src/nlohmann/json.hpp:23829

Functionbasic_json from_bson

src/nlohmann/json.hpp:23859

Functionbasic_json from_cbor

src/nlohmann/json.hpp:23663

Functionbasic_json from_msgpack

src/nlohmann/json.hpp:23721

Functionbasic_json from_ubjson

src/nlohmann/json.hpp:23775

Functionbasic_json meta

@brief returns version information on the library @sa https://json.nlohmann.me/api/basic_json/meta/

src/nlohmann/json.hpp:19549

Functionbasic_json object

@brief explicitly create an object from an initializer list @sa https://json.nlohmann.me/api/basic_json/object/

src/nlohmann/json.hpp:20314

Functionbasic_json parse

src/nlohmann/json.hpp:23316

src/nlohmann/json.hpp:6860

src/nlohmann/json.hpp:7044

Methodbinary_reader

! @brief create a binary reader @param[in] adapter input adapter to read from */

src/nlohmann/json.hpp:9228

Methodbinary_writer

! @brief create a binary writer @param[in] adapter output adapter to write to */

src/nlohmann/json.hpp:15090

Functionboolean

src/nlohmann/json.hpp:6830

src/nlohmann/json.hpp:7014

Returns the bucket number where the element with key k is located.

src/tiktoken/hash_table8.hpp:436

Returns the bucket number where the element with key k is located.

src/tiktoken/hash_set8.hpp:396

Methodbucket_count

src/tiktoken/hash_table8.hpp:416

Methodbucket_count

src/tiktoken/hash_set8.hpp:376

Methodbucket_size

Returns the number of collision elements in pos bucket.

src/tiktoken/hash_table8.hpp:449

Methodbucket_size

Returns the number of elements in bucket n.

src/tiktoken/hash_set8.hpp:409

Methodbuild_extensions

Methodcalc_bson_array_size

! @return The size of the BSON-encoded array @a value */

src/nlohmann/json.hpp:16146

Methodcalc_bson_binary_size

! @return The size of the BSON-encoded binary array @a value */

src/nlohmann/json.hpp:16161

Methodcalc_bson_element_size

! @brief Calculates the size necessary to serialize the JSON value @a j with its @a name @return The calculated size for the BSON document ent

src/nlohmann/json.hpp:16203

← previous next →201–300 of 724, ranked by callers