Functions — github.com/google/sentencepiece

CodeHub MCPcopy Create free account

hub / github.com/google/sentencepiece / functions

Functions367 in github.com/google/sentencepiece

⨍ Functions 367◇ Types & classes 60

MethodGetPrecompiledCharsMap

src/builder.cc:275

MethodGetPretokenizerForTraining

src/sentencepiece_trainer.cc:178

MethodGetPrevIndex

src/bpe_model_trainer.cc:120

src/sentencepiece_processor_test.cc:494

Returns the score of `id`. Score represents a log probability of the piece. We can roughly estimate the unigram frequency of the piece.

src/model_interface.h:106

MethodGetScoreInlined

Non-virtual (inlined) implementation for faster execution.

src/model_interface.h:138

MethodGetScriptInternal

src/unicode_script.cc:26

FunctionHexToInt

src/normalizer.cc:48

MethodInitMetaPieces

src/trainer_interface.cc:571

MethodInitializePieces

src/model_interface.cc:63

MethodIsControl

src/sentencepiece_processor_test.cc:492

MethodIsControl

Returns true if `id` is control symbol.

src/model_interface.h:117

MethodIsControlInlined

src/model_interface.h:147

MethodIsUnknown

src/sentencepiece_processor_test.cc:490

MethodIsUnknown

Returns true if `id` is unknown symbol.

src/model_interface.h:111

MethodIsUnknownInlined

src/model_interface.h:142

Returns true if `id` is unused symbol.

src/model_interface.h:123

MethodIsUnusedInlined

src/model_interface.h:152

MethodIsUserDefinedInlined

src/model_interface.h:157

src/util_test.cc:363

src/unigram_model.cc:57

MethodLoadCharsMap

src/builder.cc:485

MethodLoadOrDie

src/sentencepiece_processor.cc:59

MethodLoadSentences

src/trainer_interface.cc:260

MethodMakeSeedSentencePieces

Returns seed sentencepieces for EM training.

src/unigram_model_trainer.cc:101

MethodMergeSpecsFromArgs

src/sentencepiece_trainer.cc:68

MethodMergeUnicodeCaseFoldMap

src/builder.cc:430

MethodMockPretokenizer

src/pretokenizer_for_training_test.cc:26

src/char_model.cc:21

src/unigram_model.h:119

src/word_model.cc:21

src/bpe_model.cc:30

src/unigram_model.cc:479

MethodModelInterface

src/model_interface.h:58

MethodModelInterface

src/model_interface.cc:25

MethodNBestEncode

The same as above, but returns nbest result with score.

src/model_interface.h:77

MethodNBestEncodeAsIds

src/sentencepiece_processor.h:254

MethodNBestEncodeAsPieces

src/sentencepiece_processor.h:248

src/unigram_model.cc:83

MethodNormalizePrefix

src/normalizer.cc:193

MethodNormalizer

src/normalizer.cc:33

MethodParseExtraOptions

src/sentencepiece_processor.cc:655

MethodPieceToId

src/sentencepiece_processor_test.cc:476

MethodPieceToId

src/sentencepiece_processor.cc:564

MethodPieceToId

src/model_interface.cc:51

MethodPopulateModelTypeFromString

src/sentencepiece_trainer.cc:149

MethodPopulateNormalizerSpec

src/sentencepiece_trainer.cc:121

MethodPopulateSentencePieceText

src/sentencepiece_processor.cc:320

MethodPosixReadableFile

src/filesystem.cc:34

MethodPosixWritableFile

src/filesystem.cc:72

MethodPostprocess

src/pretokenizer_for_training.cc:44

MethodPrefixMatcher

src/normalizer.cc:286

MethodPreprocess

src/pretokenizer_for_training.cc:37

MethodPretokenizerForTrainingInterface

src/pretokenizer_for_training.h:30

MethodPruneSentencePieces

src/unigram_model_trainer.cc:280

MethodRandomGeneratorStorage

src/util.cc:138

MethodReadableFile

src/filesystem.h:31

MethodRemoveRedundantMap

src/builder.cc:552

MethodReservoirSampler

MethodResetFreq

src/bpe_model_trainer.cc:137

src/unigram_model_trainer.cc:197

src/unigram_model_trainer.cc:248

MethodSampleEncode

src/model_interface.h:83

MethodSampleEncode

src/sentencepiece_processor.cc:292

MethodSampleEncodeAsIds

src/sentencepiece_processor.h:267

MethodSampleEncodeAsPieces

src/sentencepiece_processor.h:260

src/trainer_interface.cc:565

MethodSaveCharsMap

src/builder.cc:520

MethodSaveModel

src/trainer_interface.cc:519

FunctionSaveModelProto

src/sentencepiece_processor.cc:720

MethodSaveVocab

src/trainer_interface.cc:543

MethodSentencePieceProcessor

src/sentencepiece_processor.cc:50

MethodSentencePieceTrainer

src/sentencepiece_trainer.h:92

MethodSentenceSelector

src/trainer_interface.cc:107

MethodSetPretokenizerForTraining

src/sentencepiece_trainer.cc:171

MethodSetProtoField

src/spec_parser.h:169

MethodSplitSentencesByWhitespace

src/trainer_interface.cc:453

src/deps/status.cc:7

MethodStatusBuilder

src/deps/status_builder.h:12

src/bpe_model_trainer.h:54

src/char_model_test.cc:53

src/model_factory_test.cc:22

src/model_interface_test.cc:56

src/unigram_model_test.cc:32

src/builder_test.cc:32

src/char_model_trainer_test.cc:71

src/bpe_model_test.cc:49

src/sentencepiece_processor_test.cc:115

src/bpe_model_trainer_test.cc:79

src/word_model_test.cc:54

src/filesystem_test.cc:24

src/util_test.cc:29

src/unigram_model_trainer_test.cc:33

src/pretokenizer_for_training_test.cc:41

src/freelist_test.cc:9

src/unicode_script_test.cc:31

src/sentencepiece_trainer_test.cc:39

src/trainer_interface_test.cc:28

src/trainer_factory_test.cc:22

← previous next →201–300 of 367, ranked by callers