Functions — github.com/M4THYOU/TokenDagger

CodeHub MCPcopy Create free account

hub / github.com/M4THYOU/TokenDagger / functions

Functions724 in github.com/M4THYOU/TokenDagger

⨍ Functions 724◇ Types & classes 198

↓ 3 callersMethodencode

src/main.cpp:50

↓ 3 callersMethodgenerate_article

Generate a realistic article of approximately target_length characters.

tests/performance_benchmark.py:389

↓ 3 callersFunctionis_number_float

@brief return whether value is a floating-point number @sa https://json.nlohmann.me/api/basic_json/is_number_float/

src/nlohmann/json.hpp:20652

↓ 3 callersFunctionis_number_integer

@brief return whether value is an integer number @sa https://json.nlohmann.me/api/basic_json/is_number_integer/

src/nlohmann/json.hpp:20638

↓ 3 callersMethodmax_load_factor

src/tiktoken/hash_table8.hpp:422

↓ 3 callersMethodmoved_or_copied

src/nlohmann/json.hpp:14850

↓ 3 callersMethodset_begin

set iterator to a defined beginning

src/nlohmann/json.hpp:12750

↓ 3 callersMethodtry_get

Returns the matching ValueT or nullptr if k isn't found.

src/tiktoken/hash_table8.hpp:645

↓ 3 callersFunctionunget

! @brief unget current character (read it again on next get) We implement unget by setting variable next_unget to true. The input is not

src/nlohmann/json.hpp:8746

↓ 2 callersFunctionat

@brief access specified array element with bounds checking @sa https://json.nlohmann.me/api/basic_json/at/

src/nlohmann/json.hpp:21240

↓ 2 callersMethodbenchmark_throughput

Benchmark tokenizer throughput with specified thread count.

tests/throughput_test.py:394

↓ 2 callersFunctionbyte_pair_encode

Byte pair encoding

src/tiktoken/tiktoken.cpp:371

↓ 2 callersFunctioncompute_boundaries

src/nlohmann/json.hpp:17096

↓ 2 callersMethoddecode_bytes

src/tiktoken/tiktoken.cpp:236

↓ 2 callersFunctionemplace_back

src/nlohmann/json.hpp:22511

↓ 2 callersMethodencode

src/mistral_main.cpp:63

↓ 2 callersMethodencode_ordinary

src/tiktoken/tiktoken.cpp:156

↓ 2 callersFunctionerase_internal

src/nlohmann/json.hpp:21869

↓ 2 callersFunctionfile_input_adapter

src/nlohmann/json.hpp:6193

↓ 2 callersFunctionflatten

@brief return flattened JSON value @sa https://json.nlohmann.me/api/basic_json/flatten/

src/nlohmann/json.hpp:23976

↓ 2 callersFunctionfrom_json_array_impl

src/nlohmann/json.hpp:4821

↓ 2 callersFunctionfrom_json_inplace_array_impl

src/nlohmann/json.hpp:4904

↓ 2 callersFunctionfrom_json_tuple_impl

src/nlohmann/json.hpp:5010

↓ 2 callersFunctionget_rank

src/tiktoken/tiktoken.cpp:282

↓ 2 callersFunctiongrisu2

src/nlohmann/json.hpp:17735

↓ 2 callersFunctiongrisu2_round

src/nlohmann/json.hpp:17453

↓ 2 callersFunctionhex_bytes

! * @brief convert a byte to a uppercase hex representation * @param[in] byte byte to represent * @return representation ("00".."FF")

src/nlohmann/json.hpp:18685

↓ 2 callersMethodis_end

return whether the iterator is at end

src/nlohmann/json.hpp:12768

↓ 2 callersMethodis_errored

src/nlohmann/json.hpp:7183

↓ 2 callersFunctionis_number_unsigned

@brief return whether value is an unsigned integer number @sa https://json.nlohmann.me/api/basic_json/is_number_unsigned/

src/nlohmann/json.hpp:20645

↓ 2 callersFunctionlittle_endianness

! @brief determine system byte order @return true if and only if system's byte order is little endian @note from https://stackoverflow.com/a/1001328

src/nlohmann/json.hpp:9198

↓ 2 callersMethodload_factor

src/tiktoken/hash_table8.hpp:417

↓ 2 callersMethodmax_load_factor

src/tiktoken/hash_set8.hpp:384

↓ 2 callersFunctionoperator++

! @brief post-increment (it++) @pre The iterator is initialized; i.e. `m_object != nullptr`. */

src/nlohmann/json.hpp:13234

↓ 2 callersFunctionoperator--

! @brief post-decrement (it--) @pre The iterator is initialized; i.e. `m_object != nullptr`. */

src/nlohmann/json.hpp:13285

↓ 2 callersFunctionpop_back

@brief remove last reference token @sa https://json.nlohmann.me/api/json_pointer/pop_back/

src/nlohmann/json.hpp:13964

↓ 2 callersFunctionreset

reset token_buffer; current character is beginning of token

src/nlohmann/json.hpp:8692

↓ 2 callersFunctionskip_whitespace

src/nlohmann/json.hpp:8872

↓ 2 callersMethodswap

src/tiktoken/hash_set8.hpp:338

↓ 2 callersMethodwrite_bson

! @param[in] j JSON value to serialize @pre j.type() == value_t::object */

src/nlohmann/json.hpp:15099

↓ 2 callersMethodwrite_cbor

! @param[in] j JSON value to serialize */

src/nlohmann/json.hpp:15128

↓ 2 callersMethodwrite_msgpack

! @param[in] j JSON value to serialize */

src/nlohmann/json.hpp:15452

↓ 1 callersMethodCoreBPE

src/tiktoken/tiktoken.hpp:48

↓ 1 callersFunctionLoadBPEFile

src/main.cpp:89

↓ 1 callersFunctionLoadTokenizer

src/mistral_main.cpp:99

↓ 1 callersFunctionLoadTokenizer

src/main.cpp:112

↓ 1 callersFunctionTokenize

src/main.cpp:140

↓ 1 callersMethod_load_special_tokens_file

Load special tokens from JSON file.

tokendagger/wrapper.py:129

↓ 1 callersMethod_load_vocab_file

Load vocabulary from JSON file.

tokendagger/wrapper.py:120

↓ 1 callersFunctionappend_exponent

src/nlohmann/json.hpp:17835

↓ 1 callersFunctionbase64_decode

src/mistral_main.cpp:80

↓ 1 callersFunctionbase64_decode

src/main.cpp:70

↓ 1 callersMethodbenchmark_hf_throughput

Benchmark HF Fast Tokenizer throughput.

tests/throughput_test.py:335

↓ 1 callersMethodbenchmark_single_file

Benchmark both tokenizers on a single code file.

tests/code_performance_benchmark.py:338

↓ 1 callersMethodbenchmark_single_text

Benchmark both tokenizers on a single text.

tests/performance_benchmark.py:413

↓ 1 callersFunctionbpe_merge

src/tiktoken/tiktoken.cpp:298

↓ 1 callersFunctionbuild_tiktoken_lib

Build the tiktoken static library.

build_tokendagger.py:11

↓ 1 callersFunctionclear

@brief clears the contents @sa https://json.nlohmann.me/api/basic_json/clear/

src/nlohmann/json.hpp:22329

↓ 1 callersMethodclear

Remove all elements, keeping full capacity.

src/tiktoken/hash_table8.hpp:982

↓ 1 callersMethodcompare_decoding

Compare decoding results between TokenDagger and TikToken.

tests/test_tokendagger_vs_tiktoken.py:278

↓ 1 callersMethodcompare_encoding

Compare encoding results between TokenDagger and TikToken.

tests/test_tokendagger_vs_tiktoken.py:242

↓ 1 callersFunctionconcat_into

src/nlohmann/json.hpp:4265

↓ 1 callersFunctionconcat_length

src/nlohmann/json.hpp:4234

↓ 1 callersFunctioncrbegin

@brief returns a const reverse iterator to the last element @sa https://json.nlohmann.me/api/basic_json/crbegin/

src/nlohmann/json.hpp:22148

↓ 1 callersFunctioncreate_special_tokens

tests/autogenned_test.py:36

↓ 1 callersFunctioncreate_test_vocab

tests/autogenned_test.py:10

↓ 1 callersFunctioncrend

@brief returns a const reverse iterator to one before the first @sa https://json.nlohmann.me/api/basic_json/crend/

src/nlohmann/json.hpp:22155

↓ 1 callersMethoddecode_bytes

Decode tokens to bytes. Args: tokens: List of token IDs Returns: Decoded bytes

tokendagger/wrapper.py:262

↓ 1 callersFunctiondump_float

! @brief dump a floating-point number Dump a given floating-point number to output stream @a o. Works internally with @a number_buffer.

src/nlohmann/json.hpp:18807

↓ 1 callersMethodempty

src/tiktoken/hash_set8.hpp:375

↓ 1 callersMethodencode

src/tiktoken/tiktoken.cpp:169

↓ 1 callersMethodencode_batch

Encode multiple texts in parallel. Args: text: Sequence of texts to encode num_threads: Number of threads to

tokendagger/wrapper.py:212

↓ 1 callersMethoderase

Erase an element from the hash table. return 0 if element was not found

src/tiktoken/hash_table8.hpp:900

↓ 1 callersMethodfind

src/tiktoken/hash_table8.hpp:566

↓ 1 callersMethodfind_code_files

Find all code files in the repository, excluding external libraries.

tests/code_performance_benchmark.py:268

↓ 1 callersFunctionfind_largest_pow10

! For n != 0, returns k, such that pow10 := 10^(k-1) <= n < 10^k. For n == 0, returns 1 and sets pow10 := 1. */

src/nlohmann/json.hpp:17399

↓ 1 callersFunctionformat_buffer

src/nlohmann/json.hpp:17887

↓ 1 callersMethodgenerate_performance_svg

Generate an SVG bar chart showing TokenDagger vs TikToken performance.

tests/throughput_test.py:635

↓ 1 callersMethodgenerate_test_text

Generate realistic test text of specified size in MB.

tests/throughput_test.py:246

↓ 1 callersMethodgenerate_test_texts

Generate comprehensive test corpus with various edge cases.

tests/performance_benchmark.py:239

↓ 1 callersFunctionget_cached_power_for_binary_exponent

! For a normalized diyfp w = f * 2^e, this function returns a (normalized) cached power-of-ten c = f_c * 2^e_c, such that the exponent of the product

src/nlohmann/json.hpp:17235

↓ 1 callersFunctionget_impl_ptr

src/nlohmann/json.hpp:20718

↓ 1 callersFunctionget_ptr

src/nlohmann/json.hpp:20858

↓ 1 callersMethodget_value

src/nlohmann/json.hpp:12744

↓ 1 callersFunctionget_version

↓ 1 callersFunctiongrisu2_digit_gen

! Generates V = buffer * 10^decimal_exponent, such that M- <= V <= M+. M- and M+ must be normalized and share the same exponent -60 <= e <= -32. */

src/nlohmann/json.hpp:17494

↓ 1 callersFunctionhash

src/nlohmann/json.hpp:6022

↓ 1 callersMethodinit_regex

src/tiktoken/tiktoken.cpp:48

↓ 1 callersFunctionint_to_string

src/nlohmann/json.hpp:5179

↓ 1 callersFunctionis_negative_number

src/nlohmann/json.hpp:18696

↓ 1 callersFunctionis_number

@brief return whether value is a number @sa https://json.nlohmann.me/api/basic_json/is_number/

src/nlohmann/json.hpp:20631

↓ 1 callersFunctionis_structured

@brief return whether type is structured @sa https://json.nlohmann.me/api/basic_json/is_structured/

src/nlohmann/json.hpp:20610

↓ 1 callersMethodlast

src/tiktoken/hash_table8.hpp:392

↓ 1 callersMethodlast

src/tiktoken/hash_set8.hpp:353

↓ 1 callersMethodload_bpe_vocab

Load vocabulary from tokenizer.model file.

tests/throughput_test.py:137

↓ 1 callersMethodload_bpe_vocab

Load vocabulary from tokenizer.model file.

tests/test_tokendagger_vs_tiktoken.py:107

↓ 1 callersMethodload_bpe_vocab

Load vocabulary from tokenizer.model file.

tests/code_performance_benchmark.py:215

↓ 1 callersMethodload_bpe_vocab

Load vocabulary from tokenizer.model file.

tests/performance_benchmark.py:140

↓ 1 callersMethodload_llama_config

Load Llama 4 configuration from the codebase. https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

tests/throughput_test.py:90

↓ 1 callersMethodload_llama_config

Load Llama 4 configuration from the codebase.

tests/test_tokendagger_vs_tiktoken.py:48

← previous next →101–200 of 724, ranked by callers