Functions — github.com/google/sentencepiece

CodeHub MCPcopy Create free account

hub / github.com/google/sentencepiece / functions

Functions367 in github.com/google/sentencepiece

⨍ Functions 367◇ Types & classes 60

↓ 328 callersMethodsize

src/unigram_model.cc:68

↓ 223 callersMethodok

src/deps/status.h:40

↓ 110 callersMethodNormalize

src/normalizer.cc:69

↓ 59 callersMethodIsUnused

src/sentencepiece_processor.cc:590

↓ 54 callersFunctionOkStatus

src/deps/status.h:54

↓ 53 callersMethodEncode

src/bpe_model.cc:37

↓ 42 callersFunctionAddPiece

src/unigram_model_test.cc:330

↓ 38 callersMethodsurface

src/unigram_model.cc:77

↓ 35 callersMethodstatus

Returns Status. Normalizes function is valid only when status is OK.

src/normalizer.h:78

↓ 32 callersMethodEncode

src/sentencepiece_processor_test.cc:52

↓ 26 callersMethodDecode

src/sentencepiece_processor.cc:234

↓ 24 callersFunctionAddPiece

src/bpe_model_test.cc:42

↓ 24 callersFunctionAddPiece

src/sentencepiece_processor_test.cc:611

↓ 24 callersFunctionIsStructurallyValid

↓ 24 callersMethodLoad

src/sentencepiece_processor.cc:53

↓ 23 callersMethodPieceToId

src/unigram_model.cc:425

↓ 23 callersMethodWriteLine

src/filesystem.cc:94

↓ 18 callersFunctionDecodeUTF8

↓ 18 callersMethodIsControl

src/sentencepiece_processor.cc:580

↓ 18 callersMethodSetSentence

src/unigram_model.cc:97

↓ 17 callersMethodIsUnknown

src/sentencepiece_processor.cc:585

↓ 17 callersFunctionNewWritableFile

src/filesystem.cc:110

↓ 17 callersMethodPrefixMatch

src/normalizer.cc:296

↓ 16 callersFunctionAddPiece

src/model_interface_test.cc:49

↓ 16 callersMethodIsControl

src/sentencepiece_processor_test.cc:68

↓ 16 callersMethodIsUnknown

src/sentencepiece_processor_test.cc:70

↓ 16 callersMethodPieceToId

src/sentencepiece_processor_test.cc:74

↓ 16 callersMethodSetModel

src/sentencepiece_processor.cc:689

↓ 15 callersMethodGetScore

src/sentencepiece_processor.cc:575

↓ 15 callersFunctionSplitIntoWords

src/model_interface.cc:108

↓ 15 callersFunctionUTF8ToUnicodeText

src/util.cc:109

↓ 14 callersMethodInsert

src/unigram_model.cc:132

↓ 14 callersMethodbos_piece

src/model_interface.cc:37

↓ 14 callersMethodeos_piece

src/model_interface.cc:41

↓ 14 callersMethodunk_piece

src/model_interface.cc:33

↓ 12 callersFunctionInsertWithScore

src/unigram_model_test.cc:184

↓ 12 callersFunctionInsertWithScoreAndId

src/unigram_model_test.cc:188

↓ 12 callersFunctionSorted

src/trainer_interface.h:35

↓ 12 callersFunctionUnicodeTextToUTF8

src/util.cc:122

↓ 11 callersFunctionGetScriptType

src/unicode_script_test.cc:25

↓ 11 callersFunctionNewReadableFile

src/filesystem.cc:105

↓ 11 callersMethodSetDecodeExtraOptions

src/sentencepiece_processor.cc:118

↓ 11 callersMethodSetNormalizer

src/sentencepiece_processor.cc:693

↓ 10 callersFunctionMakeBaseModelProto

src/model_interface_test.cc:32

↓ 10 callersMethodReadLine

src/filesystem.cc:51

↓ 10 callersMethodSetEncodeExtraOptions

src/sentencepiece_processor.cc:113

↓ 10 callersMethodViterbi

src/unigram_model.cc:145

↓ 9 callersFunctionContainsKey

↓ 9 callersFunctionGetSpVec

src/sentencepiece_processor_test.cc:87

↓ 9 callersFunctionGetTokenized

src/unigram_model_test.cc:176

↓ 9 callersFunctionMakeDefaultNormalizerSpec

src/sentencepiece_processor_test.cc:111

↓ 9 callersFunctionUnicodeCharToUTF8

src/util.cc:107

↓ 9 callersMethodpad_piece

src/model_interface.cc:45

↓ 8 callersMethodGetScore

src/sentencepiece_processor_test.cc:78

↓ 8 callersMethodIsUserDefined

Returns true if `id` is user defined symbol.

src/model_interface.h:129

↓ 8 callersFunctionIsValidCodepoint

↓ 8 callersFunctionMakeBaseModelProto

src/unigram_model_test.cc:314

↓ 8 callersFunctionMakeDefaultSpec

src/normalizer_test.cc:34

↓ 8 callersMethodNBestEncode

src/unigram_model.cc:518

↓ 8 callersFunctionOneCharLen

Return length of a single UTF-8 source character

↓ 8 callersMethodSampleEncode

src/unigram_model.cc:544

↓ 8 callersMethodSampleEncode

src/sentencepiece_processor_test.cc:57

↓ 8 callersMethodSetEncodeResult

src/sentencepiece_processor_test.cc:41

↓ 8 callersMethodbos_id

src/sentencepiece_processor.cc:601

↓ 8 callersMethodeos_id

src/sentencepiece_processor.cc:607

↓ 7 callersFunctionAddPiece

src/word_model_test.cc:47

↓ 7 callersFunctionCheckVocab

src/sentencepiece_trainer_test.cc:31

↓ 7 callersMethodpad_id

src/sentencepiece_processor.cc:613

↓ 6 callersFunctionAddPiece

src/char_model_test.cc:46

↓ 6 callersMethodAllocate

Allocates new element.

src/freelist.h:54

↓ 6 callersMethodGetPieceSize

src/sentencepiece_processor.cc:559

↓ 6 callersMethodGlobalReplace

src/normalizer.cc:320

↓ 6 callersFunctionIsTrailByte

Return (x & 0xC0) == 0x80; Since trail bytes are always in [0x80, 0xBF], we can optimize:

↓ 6 callersFunctionItoa

↓ 6 callersFunctionJoinPath

↓ 6 callersMethodLoadVocabulary

src/sentencepiece_processor.cc:171

↓ 6 callersMethodPopulateNodes

src/unigram_model.cc:373

↓ 6 callersMethodToString

src/bpe_model_trainer.cc:27

↓ 6 callersMethodunk_id

src/sentencepiece_processor.cc:595

↓ 6 callersMethodutf8_size

src/unigram_model.cc:73

↓ 5 callersFunctionEncodeUTF8

↓ 5 callersFunctionGetIdVec

src/sentencepiece_processor_test.cc:95

↓ 5 callersMethodIgnoreError

src/deps/status.cc:114

↓ 5 callersMethodIsValidSentencePiece

src/trainer_interface.cc:176

↓ 5 callersMethodNBest

src/unigram_model.cc:228

↓ 5 callersMethodWrite

src/filesystem.cc:89

↓ 4 callersFunctionMakeBaseModelProto

src/bpe_model_test.cc:26

↓ 4 callersMethodNBestEncode

src/sentencepiece_processor_test.cc:62

↓ 4 callersFunctionRunTrainer

src/bpe_model_trainer_test.cc:36

↓ 4 callersMethodSetNBestEncodeResult

src/sentencepiece_processor_test.cc:46

↓ 4 callersMethodTrain

src/bpe_model_trainer.cc:174

↓ 4 callersFunctionUnicodeNormalize

Normalize `input` with ICU's normalizer with `mode`.

src/builder.cc:53

↓ 4 callersMethodsentence

src/unigram_model.cc:75

↓ 3 callersFunctionIntToHex

↓ 3 callersFunctionLogSumExp

Returns log(exp(x) + exp(y)). if init_mode is true, returns log(exp(y)) == y. log(\sum_i exp(a[i])) can be computed as for (int i = 0; i < a.size(); +

src/unigram_model.cc:42

↓ 3 callersFunctionNormalize

Normalizes `src` with `chars_map` and returns normalized Chars. `max_len` specifies the maximum length of the key in `chars_map`.

src/builder.cc:122

↓ 3 callersMethodPreTokenize

src/pretokenizer_for_training.cc:31

↓ 3 callersMethodSchedule

src/deps/threadpool.h:18

↓ 3 callersMethodSerialize

src/trainer_interface.cc:467

↓ 3 callersMethodSetSentencePieces

src/unigram_model_trainer.cc:76

next →1–100 of 367, ranked by callers