MCPcopy Create free account

hub / github.com/NLPOptimize/flash-tokenizer / functions

Functions146 in github.com/NLPOptimize/flash-tokenizer

↓ 24 callersMethodsize
include/robin_hood.h:2105
↓ 8 callersMethodempty
include/robin_hood.h:2115
↓ 7 callersFunctionswap
include/robin_hood.h:660
↓ 6 callersMethodgetFirst
include/robin_hood.h:996
↓ 5 callersMethodend
include/robin_hood.h:2010
↓ 5 callersMethodgetSecond
include/robin_hood.h:1019
↓ 4 callersMethodcalcNumElementsWithBuffer
include/robin_hood.h:2152
↓ 4 callersMethodclear
Clears all data, without resizing.
include/robin_hood.h:1682
↓ 4 callersMethoddestroy
include/robin_hood.h:2464
↓ 4 callersFunctionhash_int
include/robin_hood.h:748
↓ 4 callersMethodswap
include/robin_hood.h:648
↓ 3 callersMethodbegin
include/robin_hood.h:1991
↓ 3 callersMethodreserve
reserves space for the specified number of elements. Makes sure the old data fits. Exactly the same as rehash(c). Use rehash(0) to shrink to fit.
include/robin_hood.h:2079
↓ 3 callersMethodtokenize
include/bert_tokenizer.h:415
↓ 2 callersMethodfind
include/robin_hood.h:1949
↓ 2 callersFunctionhash_bytes
include/robin_hood.h:692
↓ 2 callersMethodinit
include/robin_hood.h:2483
↓ 2 callersMethodnodes
include/robin_hood.h:1179
↓ 1 callersMethodallocate
allocates, but does NOT initialize. Use in-place new constructor, e.g. T* obj = pool.allocate(); ::new (static_cast<void*>(obj)) T();
include/robin_hood.h:417
↓ 1 callersMethodcalcNumBytesInfo
include/robin_hood.h:2145
↓ 1 callersMethodcalcNumBytesTotal
include/robin_hood.h:2158
↓ 1 callersFunctioncheck_platform
main.cpp:15
↓ 1 callersFunctionclean_text
include/bert_tokenizer.h:151
↓ 1 callersMethodconvert_tokens_to_ids
include/bert_tokenizer.h:512
↓ 1 callersMethodcount
Returns 1 if key is found, 0 otherwise.
include/robin_hood.h:1893
↓ 1 callersMethoddeallocate
does not actually deallocate but puts it in store. make sure you have already called the destructor! e.g. with obj->~T(); pool.deallocate(obj);
include/robin_hood.h:431
↓ 1 callersMethoddestroyDoNotDeallocate
include/robin_hood.h:977
↓ 1 callersMethoderase
include/robin_hood.h:2025
↓ 1 callersFunctionload_gt
main.cpp:61
↓ 1 callersFunctionload_titles
main.cpp:49
↓ 1 callersMethodnodesDoNotDeallocate
include/robin_hood.h:1183
↓ 1 callersFunctionparseNumbersFromString
main.cpp:35
↓ 1 callersFunctionsplit_on_punc
include/bert_tokenizer.h:227
↓ 1 callersFunctiontest
main.cpp:73
↓ 1 callersFunctionto_lower
include/bert_tokenizer.h:202
↓ 1 callersFunctionto_lower_ascii
include/bert_tokenizer.h:198
↓ 1 callersMethodtokenize
include/bert_tokenizer.h:494
MethodBasicTokenizer
include/bert_tokenizer.h:394
MethodBulkPoolAllocator
include/robin_hood.h:369
MethodDataNode
include/robin_hood.h:967
MethodDataNode
include/robin_hood.h:1043
MethodFlashBertTokenizer
include/bert_tokenizer.h:486
MethodIter
default constructed iterator can be compared to itself, but WON'T return true when compared to end().
include/robin_hood.h:1238
FunctionPYBIND11_MODULE
python/src/bindings.cpp:10
FunctionSTRING_LIST to_lower_split_on_punc
include/bert_tokenizer.h:273
FunctionSTRING_LIST whitespace_tokenize
include/bert_tokenizer.h:130
MethodTable
include/robin_hood.h:1497
FunctionVocab
include/bert_tokenizer.h:93
MethodWordpieceTokenizer
include/bert_tokenizer.h:442
MethodWrapHash
include/robin_hood.h:869
MethodWrapKeyEqual
include/robin_hood.h:876
Methodadd
WARNING: Underflow if numBytes < ALIGNMENT! This is guarded in addOrFree().
include/robin_hood.h:476
MethodaddOrFree
Adds an already allocated block of memory to the allocator. This allocator is from now on responsible for freeing the data (with free()). If the provi
include/robin_hood.h:439
MethodaddOrFree
we are not using the data, so just free it.
include/robin_hood.h:548
FunctionassertNotNull
include/robin_hood.h:347
Methodat
NOLINTNEXTLINE(modernize-use-nodiscard)
include/robin_hood.h:1927
MethodcalcMaxNumElementsAllowed
include/robin_hood.h:2136
MethodcalcNumElementsToAlloc
include/robin_hood.h:462
Methodcbegin
include/robin_hood.h:2002
Methodcend
include/robin_hood.h:2020
MethodcloneData
include/robin_hood.h:1444
Methodcompact
If possible reallocates the map to a smaller one. This frees the underlying table. Does not do anything if load_factor is too large for decreasing the
include/robin_hood.h:2086
Methodcontains
include/robin_hood.h:1913
Methodconvert_ids_to_tokens
include/bert_tokenizer.h:517
Methoddestroy
doesn't do anything
include/robin_hood.h:976
Methoddestroy
include/robin_hood.h:1051
MethoddestroyDoNotDeallocate
include/robin_hood.h:1057
FunctiondoThrow
include/robin_hood.h:336
Methodemplace
include/robin_hood.h:1796
Methodemplace_hint
include/robin_hood.h:1824
MethodfastForward
fast forward to the next non-free info byte I've tried a few variants that don't depend on intrinsics, but unfortunately they are quite a bit slower t
include/robin_hood.h:1306
MethodfindIdx
include/robin_hood.h:1419
MethodgetFirst
include/robin_hood.h:1079
MethodgetFirstConst
include/robin_hood.h:1137
MethodgetSecond
include/robin_hood.h:1102
Methodhas
include/robin_hood.h:2180
Methodincrease_size
True if resize was possible, false otherwise
include/robin_hood.h:2428
MethodinitData
include/robin_hood.h:2318
Methodinsert
include/robin_hood.h:1782
MethodinsertKeyPrepareEmptySpot
include/robin_hood.h:2347
MethodinsertOrAssignImpl
include/robin_hood.h:2289
Methodinsert_move
inserts a keyval that is guaranteed to be new, e.g. when the hashmap is resized. @return True on success, false if something went wrong
include/robin_hood.h:1450
Methodinsert_or_assign
include/robin_hood.h:1852
MethodisChineseChar
include/bert_tokenizer.h:323
MethodkeyToIdx
include/robin_hood.h:1348
Methodload_factor
Average number of elements per bucket. Since we allow only 1 per bucket
include/robin_hood.h:2126
Functionmain
main.cpp:114
Methodmask
include/robin_hood.h:2131
Methodmax_load_factor
include/robin_hood.h:2120
Methodmax_size
include/robin_hood.h:2110
Methodnext
forwards the index by one, wrapping around at the end
include/robin_hood.h:1363
MethodnextHashMultiplier
include/robin_hood.h:2458
MethodnextWhileLess
include/robin_hood.h:1368
Methodnodes
include/robin_hood.h:1190
MethodnodesDoNotDeallocate
include/robin_hood.h:1204
Functionoperator!=
include/robin_hood.h:670
Methodoperator!=
include/robin_hood.h:1298
Methodoperator!=
include/robin_hood.h:1723
Methodoperator()
include/robin_hood.h:764
Methodoperator()
include/robin_hood.h:775
next →1–100 of 146, ranked by callers