MCPcopy Create free account

hub / github.com/alasdairforsythe/tokenmonster / functions

Functions381 in github.com/alasdairforsythe/tokenmonster

⨍ Functions 381◇ Types & classes 47
↓ 148 callersMethodsize
tokenmonster-cpp/include/tokenmonster/tokenmonster.hpp:106
↓ 60 callersFunctionmax_zero_and
tokenmonster-cpp/src/tokenmonster.cpp:30
↓ 35 callersMethodlongest_substring
tokenmonster-cpp/src/tokenmonster.cpp:786
↓ 35 callersFunctionmin_int
tokenmonster-cpp/src/tokenmonster.cpp:28
↓ 28 callersFunctiondecodeRune
(b []byte)
training/getalltokens.go:331
↓ 26 callersMethodresize
Changes the size of the vocabulary and optionally resets the token IDs. A vocabulary can be enlarged as well reduced in size. Only t
python/tokenmonster.py:823
↓ 25 callersFunctionmax_int
tokenmonster-cpp/src/tokenmonster.cpp:29
↓ 25 callersFunctionnew
Creates a new vocabulary from a YAML string. A sample YAML file can be found here: https://github.com/alasdairforsythe/tokenmonster/yaml_guid
python/tokenmonster.py:58
↓ 25 callersFunctionsendError
(statusCode byte)
training/tokenmonsterserver.go:104
↓ 24 callersMethodfind
tokenmonster-cpp/src/tokenmonster.cpp:699
↓ 21 callersMethodLen
Returns number of tokens in the vocabulary, inluding UNK token if it is used.
go/tokenmonster.go:2513
↓ 20 callersFunctionequal_to
tokenmonster-cpp/src/tokenmonster.cpp:32
↓ 20 callersFunctionless_than
tokenmonster-cpp/src/tokenmonster.cpp:31
↓ 20 callersFunctionwrite_byte
tokenmonster-cpp/tests/unit.cpp:12
↓ 19 callersFunctionisLetter
(r rune)
training/getalltokens.go:298
↓ 19 callersFunctionwriteUint32
(buf []byte, v uint32)
training/tokenmonsterserver.go:88
↓ 18 callersMethodPrivateGenerateVocab
Don't use this function, it's exported because it's used by the exportvocab tool.
go/tokenmonster.go:2898
↓ 17 callersMethod_communicate
(self, job_type, id, data_length, data = None)
python/tokenmonster.py:1036
↓ 14 callersMethodDecode
Decodes tokens backs into bytes. If you are decoding a stream of tokens individually or in batches, instead of all at once, you should use the Decode
go/tokenmonster.go:788
↓ 14 callersMethodNormalize
--------- TOKENIZE --------- Applies all normalizations to the bytes, including capcode and NFD.
go/tokenmonster.go:953
↓ 14 callersFunctiondie
(msg string, showUsage bool)
training/exportvocab.go:96
↓ 14 callersFunctionformatInt
(v int)
training/trainvocab.go:153
↓ 14 callersFunctionprint_hex
tokenmonster-cpp/tests/dump.cpp:15
↓ 13 callersFunction_read_uint32
(input)
python/tokenmonster.py:1281
↓ 13 callersFunctionisCapcode
(r rune)
training/getalltokens.go:302
↓ 12 callersFunctionisLetter
(r rune, usingCapcode uint8)
go/tokenmonster.go:359
↓ 12 callersFunctionisLetter
(r rune)
training/trainvocab.go:253
↓ 12 callersMethodread_byte
tokenmonster-cpp/src/tokenmonster.cpp:139
↓ 11 callersMethoddecode
tokenmonster-cpp/src/tokenmonster.cpp:1420
↓ 10 callersFunction_write_uint64
(input)
python/tokenmonster.py:1275
↓ 10 callersFunctionreadUint32
(buf []byte)
training/tokenmonsterserver.go:69
↓ 10 callersMethodread_uint24
tokenmonster-cpp/src/tokenmonster.cpp:144
↓ 10 callersFunctionunleak
(b []byte)
go/tokenmonster.go:432
↓ 10 callersFunctionwriteUint64
(buf []byte, v uint64)
training/tokenmonsterserver.go:73
↓ 9 callersMethod_file_exists
(cls, fname)
python/tokenmonster.py:1033
↓ 9 callersFunction_write_uint8
(input)
python/tokenmonster.py:1278
↓ 9 callersFunctionis_iterable
(obj)
python/tokenmonster.py:1290
↓ 9 callersFunctionnormalize
(data []byte, usingCapcode uint8, normalizer norm.Normalizer)
go/tokenmonster.go:242
↓ 9 callersFunctionwrite_uint24
tokenmonster-cpp/tests/unit.cpp:16
↓ 8 callersFunctionString
(b []byte)
training/trainvocab.go:157
↓ 8 callersFunction_write_uint32
(input)
python/tokenmonster.py:1272
↓ 8 callersFunctionbytes
tokenmonster-cpp/tests/bench.cpp:22
↓ 8 callersFunctionprocessChunkMulti
(asset workStruct, numChunks int, trim bool, level uint8)
training/getalltokens.go:1236
↓ 8 callersMethodset
(key, value)
javascript/tokenmonster.js:36
↓ 8 callersMethodtokenize
Tokenizes a string into tokens according to the vocabulary. You can pass a string or a list of strings. If you pass a list of string
python/tokenmonster.py:410
↓ 7 callersFunctiondecodeLastRune
(b []byte)
training/getalltokens.go:361
↓ 7 callersFunctiondecodeRune
(b []byte)
training/trainvocab.go:265
↓ 7 callersMethodfindLargestSubarray
(arr)
javascript/tokenmonster.js:68
↓ 7 callersFunctionloadTokensFromFile
(filename string)
training/trainvocab.go:453
↓ 7 callersMethodmodify
Modifies the vocabulary. Doing so invalidates all decoder objects associated with the model before modification. Notes:
python/tokenmonster.py:674
↓ 7 callersMethodnormalize
tokenmonster-cpp/src/tokenmonster.cpp:1361
↓ 6 callersFunction_read_uint64
(input)
python/tokenmonster.py:1284
↓ 6 callersMethoddecode
(data)
javascript/tokenmonster.js:1015
↓ 6 callersFunctiondecodeRune
(b []byte, charsetFlag uint8)
go/tokenmonster.go:371
↓ 6 callersFunctionisAlphaNum
(r rune)
training/trainvocab.go:257
↓ 6 callersFunctionreadUint64
(buf []byte)
training/tokenmonsterserver.go:65
↓ 5 callersFunctioncontainsOnlyNumbers
(input string)
training/getalltokens.go:1604
↓ 5 callersMethoddecode
Decodes tokens into a string. Only use this "decode" method if you are decoding a complete "batch" or complete "conversation".
python/tokenmonster.py:341
↓ 5 callersMethodflush
tokenmonster-cpp/src/tokenmonster.cpp:1717
↓ 5 callersFunctionisAlphaNum
(r rune, usingCapcode uint8)
go/tokenmonster.go:363
↓ 5 callersFunctionisDelimiter
* func isDelimiter(r rune) bool { if r == '(' || r == ')' || r == '[' || r == ']' || r == '{' || r == '}' || r == '\'' || r == '"' || r == '‘' || r =
training/getalltokens.go:323
↓ 5 callersFunctionisLower
(r)
javascript/tokenmonster.js:884
↓ 5 callersFunctionisModifier
(r)
javascript/tokenmonster.js:896
↓ 5 callersFunctionisNumber
(r)
javascript/tokenmonster.js:892
↓ 5 callersFunctionnormalize
(data []byte)
training/trainvocab.go:335
↓ 5 callersFunctionsaveTokensToFile
(filename string, data [][]byte, data2 [][]byte, data3 [][]byte, scores []uint32, datasize int, special [][]by
training/trainvocab.go:412
↓ 5 callersMethodtokenize
tokenmonster-cpp/src/tokenmonster.cpp:3152
↓ 5 callersMethodtokenize_serialized
tokenmonster-cpp/src/tokenmonster.cpp:3164
↓ 4 callersMethodCapcode
The capcode level. 0 = disabled, 1 = deleteToken only, 2 = fully enabled.
go/tokenmonster.go:2561
↓ 4 callersMethodCharset
The charset code for the vocabulary. 0 = None, 1 = UTF-8, 2 = UTF-16.
go/tokenmonster.go:2555
↓ 4 callersMethodHighestTokenID
Returns the value of the highest token ID.
go/tokenmonster.go:2595
↓ 4 callersMethodMode
The original filter for training the vocabulary. 0 = unfiltered, 1 = clean, 2 = balanced, 3 = consistent, 4 = strict, 5 = not trained with trainvocab.
go/tokenmonster.go:2567
↓ 4 callersMethod_addToBloom
(bloomFilter, key, length)
javascript/tokenmonster.js:26
↓ 4 callersMethod_bytes_to_string
(self, input)
python/tokenmonster.py:944
↓ 4 callersMethod_checkBloom
(bloomFilter, key, length)
javascript/tokenmonster.js:31
↓ 4 callersMethod_hash
(key, length = key.length)
javascript/tokenmonster.js:10
↓ 4 callersMethod_string_to_bytes
(self, input)
python/tokenmonster.py:952
↓ 4 callersFunctionbytes
tokenmonster-cpp/tests/unit.cpp:81
↓ 4 callersMethoddecode_serialized
tokenmonster-cpp/src/tokenmonster.cpp:1504
↓ 4 callersFunctionescapeYAML
(b *bytes.Buffer, s []byte)
go/tokenmonster.go:3970
↓ 4 callersFunctionflagRequired
(name string, value interface{})
training/trainvocab.go:126
↓ 4 callersMethodget_dictionary
Returns a dictionary of all tokens in the vocabulary. This returns a list of dictionaries with keys "id", "token", "token_decoded",
python/tokenmonster.py:555
↓ 4 callersFunctionincompleteUTF16Bytes
(bytes []byte)
go/tokenmonster.go:193
↓ 4 callersFunctionincompleteUTF8Bytes
* 'flag' bits: 1 ends with a letter 2 begins with a letter 4 begins with a space OR characterToken OR wordToken 8 ends on capcode 16 begins on
go/tokenmonster.go:149
↓ 4 callersFunctionincomplete_utf16_bytes
tokenmonster-cpp/src/tokenmonster.cpp:112
↓ 4 callersFunctionincomplete_utf8_bytes
tokenmonster-cpp/src/tokenmonster.cpp:83
↓ 4 callersFunctionisAlphaNum
(r rune)
training/getalltokens.go:310
↓ 4 callersFunctionisLetter
(r)
javascript/tokenmonster.js:888
↓ 4 callersFunctionload
Loads a TokenMonster vocabulary from file, URL or by name. Parameters: path (string): A filepath, URL or pre-built vocabulary name.
python/tokenmonster.py:25
↓ 4 callersFunctionnfd
tokenmonster-cpp/src/tokenmonster.cpp:190
↓ 4 callersFunctionrun_bench
tokenmonster-cpp/tests/bench.cpp:57
↓ 4 callersFunctionstripLastPunc
* func stripLastPunc(tok []byte) []byte { rlast, nlast := decodeLastRune(tok) if isOther(rlast) { if unicode.IsSpace(rlast) || isDelimiter(rlast)
training/getalltokens.go:426
↓ 4 callersFunctionto_filename
(s)
training/dataset_helpers/download_code_samples.py:19
↓ 4 callersFunctionwriteFloatPrintable
(writer io.Writer, value float32)
go/tokenmonster.go:4010
↓ 4 callersFunctionwrite_token
tokenmonster-cpp/tests/unit.cpp:40
↓ 3 callersMethodNormalizationCode
The type of normalization applied automatically when tokenizing. Returns a uint8.
go/tokenmonster.go:2579
↓ 3 callersMethod_connect
(cls)
python/tokenmonster.py:1136
↓ 3 callersMethod_disconnect
(cls)
python/tokenmonster.py:1018
↓ 3 callersMethod_download
(cls, url, fname)
python/tokenmonster.py:1029
↓ 3 callersMethod_format_list
(self, data)
python/tokenmonster.py:960
next →1–100 of 381, ranked by callers