MCPcopy Create free account

hub / github.com/alasdairforsythe/tokenmonster / types & classes

Types & classes47 in github.com/alasdairforsythe/tokenmonster

⨍ Functions 381◇ Types & classes 47
↓ 17 callersClassError
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:21
↓ 2 callersClassVocab
Main class for TokenMonster. Usage: vocab = tokenmonster.Load("english-32000-balanced-v1") tokens = vocab.tokenize(str)
python/tokenmonster.py:107
↓ 1 callersClassDecoder
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:52
↓ 1 callersClassVocab
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:50
Class
javascript/tokenmonster.js:1
Class
javascript/tokenmonster.js:182
Class
javascript/tokenmonster.js:252
Class
javascript/tokenmonster.js:1007
ClassCapcodeDecoder
javascript/tokenmonster.js:1007
ClassCountResult
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:31
ClassDecoder
javascript/tokenmonster.js:252
StructDecoder
A decoder object for sequential decoding. Use the NewDecoder function of the Vocab struct.
go/tokenmonster.go:57
ClassDecoderInstance
A nested class for decoding streams of tokens in sequence. This class takes tokens and decodes them to generate human-readable strin
python/tokenmonster.py:117
ClassDeletedToken
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:139
ClassDoubleHash
tokenmonster-cpp/src/tokenmonster.cpp:907
ClassFastDictionary
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:145
ClassInfo
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:42
StructInfo
--------- GENERAL FUNCTIONS --------- Info struct allows access to detailed information about each token from TokensDetailed(). Token is the token sti
go/tokenmonster.go:2368
StructJsonData
Parse JSON
training/trainvocab.go:1465
ClassKeyVal
tokenmonster-cpp/src/tokenmonster.cpp:902
ClassLongestResult
tokenmonster-cpp/src/tokenmonster.cpp:493
ClassReader
tokenmonster-cpp/src/tokenmonster.cpp:131
ClassRune
tokenmonster-cpp/src/tokenmonster.cpp:38
ClassSerializedResult
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:36
ClassTokenInfo
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:133
ClassTokenInner
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:117
ClassTokenMonster
javascript/tokenmonster.js:182
ClassTokenOuter
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:122
ClassTokenizeResult
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:26
ClassUint8ArrayHashMap
javascript/tokenmonster.js:1
StructVocab
The main struct for the vocabulary
go/tokenmonster.go:38
ClassVocab::FastDictionary
tokenmonster-cpp/src/tokenmonster.cpp:491
StructYamlItem
go/tokenmonster.go:3824
StructYamlVocab
-------- YAML parsing --------
go/tokenmonster.go:3805
StructbestStruct
training/trainvocab.go:96
StructdeletedStruct
go/tokenmonster.go:85
StructresultStruct
training/trainvocab.go:80
StructtokenInfo
go/tokenmonster.go:63
StructtokenInfo
training/trainvocab.go:101
StructtokenInner
go/tokenmonster.go:80
StructtokenInner
training/trainvocab.go:116
StructtokenOuter
go/tokenmonster.go:69
StructtokenOuter
training/trainvocab.go:105
Structwork
training/tokenmonsterserver.go:53
StructworkCount
training/tokenmonsterserver.go:58
StructworkStruct
training/getalltokens.go:95
StructworkStruct
training/trainvocab.go:90