Code
Hub
Workspaces
Following
Trending
Connect
MCP
copy
Create free account
hub
/
github.com/NLPOptimize/flash-tokenizer
/ functions
Functions
146 in github.com/NLPOptimize/flash-tokenizer
⨍
Functions
146
◇
Types & classes
45
↓ 24 callers
Method
size
include/robin_hood.h:2105
↓ 8 callers
Method
empty
include/robin_hood.h:2115
↓ 7 callers
Function
swap
include/robin_hood.h:660
↓ 6 callers
Method
getFirst
include/robin_hood.h:996
↓ 5 callers
Method
end
include/robin_hood.h:2010
↓ 5 callers
Method
getSecond
include/robin_hood.h:1019
↓ 4 callers
Method
calcNumElementsWithBuffer
include/robin_hood.h:2152
↓ 4 callers
Method
clear
Clears all data, without resizing.
include/robin_hood.h:1682
↓ 4 callers
Method
destroy
include/robin_hood.h:2464
↓ 4 callers
Function
hash_int
include/robin_hood.h:748
↓ 4 callers
Method
swap
include/robin_hood.h:648
↓ 3 callers
Method
begin
include/robin_hood.h:1991
↓ 3 callers
Method
reserve
reserves space for the specified number of elements. Makes sure the old data fits. Exactly the same as rehash(c). Use rehash(0) to shrink to fit.
include/robin_hood.h:2079
↓ 3 callers
Method
tokenize
include/bert_tokenizer.h:415
↓ 2 callers
Method
find
include/robin_hood.h:1949
↓ 2 callers
Function
hash_bytes
include/robin_hood.h:692
↓ 2 callers
Method
init
include/robin_hood.h:2483
↓ 2 callers
Method
nodes
include/robin_hood.h:1179
↓ 1 callers
Method
allocate
allocates, but does NOT initialize. Use in-place new constructor, e.g. T* obj = pool.allocate(); ::new (static_cast<void*>(obj)) T();
include/robin_hood.h:417
↓ 1 callers
Method
calcNumBytesInfo
include/robin_hood.h:2145
↓ 1 callers
Method
calcNumBytesTotal
include/robin_hood.h:2158
↓ 1 callers
Function
check_platform
main.cpp:15
↓ 1 callers
Function
clean_text
include/bert_tokenizer.h:151
↓ 1 callers
Method
convert_tokens_to_ids
include/bert_tokenizer.h:512
↓ 1 callers
Method
count
Returns 1 if key is found, 0 otherwise.
include/robin_hood.h:1893
↓ 1 callers
Method
deallocate
does not actually deallocate but puts it in store. make sure you have already called the destructor! e.g. with obj->~T(); pool.deallocate(obj);
include/robin_hood.h:431
↓ 1 callers
Method
destroyDoNotDeallocate
include/robin_hood.h:977
↓ 1 callers
Method
erase
include/robin_hood.h:2025
↓ 1 callers
Function
load_gt
main.cpp:61
↓ 1 callers
Function
load_titles
main.cpp:49
↓ 1 callers
Method
nodesDoNotDeallocate
include/robin_hood.h:1183
↓ 1 callers
Function
parseNumbersFromString
main.cpp:35
↓ 1 callers
Function
split_on_punc
include/bert_tokenizer.h:227
↓ 1 callers
Function
test
main.cpp:73
↓ 1 callers
Function
to_lower
include/bert_tokenizer.h:202
↓ 1 callers
Function
to_lower_ascii
include/bert_tokenizer.h:198
↓ 1 callers
Method
tokenize
include/bert_tokenizer.h:494
Method
BasicTokenizer
include/bert_tokenizer.h:394
Method
BulkPoolAllocator
include/robin_hood.h:369
Method
DataNode
include/robin_hood.h:967
Method
DataNode
include/robin_hood.h:1043
Method
FlashBertTokenizer
include/bert_tokenizer.h:486
Method
Iter
default constructed iterator can be compared to itself, but WON'T return true when compared to end().
include/robin_hood.h:1238
Function
PYBIND11_MODULE
python/src/bindings.cpp:10
Function
STRING_LIST to_lower_split_on_punc
include/bert_tokenizer.h:273
Function
STRING_LIST whitespace_tokenize
include/bert_tokenizer.h:130
Method
Table
include/robin_hood.h:1497
Function
Vocab
include/bert_tokenizer.h:93
Method
WordpieceTokenizer
include/bert_tokenizer.h:442
Method
WrapHash
include/robin_hood.h:869
Method
WrapKeyEqual
include/robin_hood.h:876
Method
add
WARNING: Underflow if numBytes < ALIGNMENT! This is guarded in addOrFree().
include/robin_hood.h:476
Method
addOrFree
Adds an already allocated block of memory to the allocator. This allocator is from now on responsible for freeing the data (with free()). If the provi
include/robin_hood.h:439
Method
addOrFree
we are not using the data, so just free it.
include/robin_hood.h:548
Function
assertNotNull
include/robin_hood.h:347
Method
at
NOLINTNEXTLINE(modernize-use-nodiscard)
include/robin_hood.h:1927
Method
calcMaxNumElementsAllowed
include/robin_hood.h:2136
Method
calcNumElementsToAlloc
include/robin_hood.h:462
Method
cbegin
include/robin_hood.h:2002
Method
cend
include/robin_hood.h:2020
Method
cloneData
include/robin_hood.h:1444
Method
compact
If possible reallocates the map to a smaller one. This frees the underlying table. Does not do anything if load_factor is too large for decreasing the
include/robin_hood.h:2086
Method
contains
include/robin_hood.h:1913
Method
convert_ids_to_tokens
include/bert_tokenizer.h:517
Method
destroy
doesn't do anything
include/robin_hood.h:976
Method
destroy
include/robin_hood.h:1051
Method
destroyDoNotDeallocate
include/robin_hood.h:1057
Function
doThrow
include/robin_hood.h:336
Method
emplace
include/robin_hood.h:1796
Method
emplace_hint
include/robin_hood.h:1824
Method
fastForward
fast forward to the next non-free info byte I've tried a few variants that don't depend on intrinsics, but unfortunately they are quite a bit slower t
include/robin_hood.h:1306
Method
findIdx
include/robin_hood.h:1419
Method
getFirst
include/robin_hood.h:1079
Method
getFirstConst
include/robin_hood.h:1137
Method
getSecond
include/robin_hood.h:1102
Method
has
include/robin_hood.h:2180
Method
increase_size
True if resize was possible, false otherwise
include/robin_hood.h:2428
Method
initData
include/robin_hood.h:2318
Method
insert
include/robin_hood.h:1782
Method
insertKeyPrepareEmptySpot
include/robin_hood.h:2347
Method
insertOrAssignImpl
include/robin_hood.h:2289
Method
insert_move
inserts a keyval that is guaranteed to be new, e.g. when the hashmap is resized. @return True on success, false if something went wrong
include/robin_hood.h:1450
Method
insert_or_assign
include/robin_hood.h:1852
Method
isChineseChar
include/bert_tokenizer.h:323
Method
keyToIdx
include/robin_hood.h:1348
Method
load_factor
Average number of elements per bucket. Since we allow only 1 per bucket
include/robin_hood.h:2126
Function
main
main.cpp:114
Method
mask
include/robin_hood.h:2131
Method
max_load_factor
include/robin_hood.h:2120
Method
max_size
include/robin_hood.h:2110
Method
next
forwards the index by one, wrapping around at the end
include/robin_hood.h:1363
Method
nextHashMultiplier
include/robin_hood.h:2458
Method
nextWhileLess
include/robin_hood.h:1368
Method
nodes
include/robin_hood.h:1190
Method
nodesDoNotDeallocate
include/robin_hood.h:1204
Function
operator!=
include/robin_hood.h:670
Method
operator!=
include/robin_hood.h:1298
Method
operator!=
include/robin_hood.h:1723
Method
operator()
include/robin_hood.h:764
Method
operator()
include/robin_hood.h:775
next →
1–100 of 146, ranked by callers