Functions — github.com/google/sentencepiece

CodeHub MCPcopy Create free account

hub / github.com/google/sentencepiece / functions

Functions367 in github.com/google/sentencepiece

⨍ Functions 367◇ Types & classes 60

↓ 3 callersMethodSetVocabulary

src/sentencepiece_processor.cc:131

↓ 3 callersMethodStartWorkers

src/deps/threadpool.h:17

↓ 3 callersMethodbos_node

src/unigram_model.cc:79

↓ 3 callersMethodeos_node

src/unigram_model.cc:81

↓ 3 callersMethodstatus

src/sentencepiece_processor.cc:123

↓ 2 callersMethodAdd

src/trainer_interface.cc:135

↓ 2 callersMethodAdd

↓ 2 callersMethodClear

src/unigram_model.cc:89

↓ 2 callersFunctionDecodeUTF8

mblen sotres the number of bytes consumed after decoding.

↓ 2 callersFunctionDigamma

src/unigram_model_trainer.cc:41

↓ 2 callersMethodFree

`Free` doesn't free the object but reuse the allocated memory chunks.

src/freelist.h:35

↓ 2 callersMethodGetPieceSize

src/sentencepiece_processor_test.cc:72

↓ 2 callersFunctionGetRandomGenerator

src/util.cc:158

↓ 2 callersFunctionGetScript

src/unicode_script.cc:37

↓ 2 callersFunctionInsertIfNotPresent

↓ 2 callersFunctionInsertOrDie

↓ 2 callersMethodLoadFromSerializedProto

src/sentencepiece_processor.cc:69

↓ 2 callersFunctionMakeBaseModelProto

src/char_model_test.cc:30

↓ 2 callersFunctionMakeBaseModelProto

src/word_model_test.cc:31

↓ 2 callersMethodPopulateMarginal

src/unigram_model.cc:181

↓ 2 callersFunctionPrintProto

src/spec_parser.h:100

↓ 2 callersFunctionRunTrainer

src/char_model_trainer_test.cc:35

↓ 2 callersFunctionSTLDeleteElements

↓ 2 callersMethodSample

src/unigram_model.cc:329

↓ 2 callersMethodSetOutput

src/pretokenizer_for_training_test.cc:35

↓ 2 callersFunctionStrError

src/util.cc:172

↓ 2 callersMethodmin_score

Returns the minimum score in sentence pieces. min_score() - 10 is used for the cost of unknown sentence.

src/unigram_model.h:132

↓ 2 callersMethodsize

Returns the number of allocated elements.

src/freelist.h:46

↓ 2 callersMethodtotal_size

src/trainer_interface.cc:156

↓ 1 callersMethodDecodePiecesAsSerializedProto

src/sentencepiece_processor.cc:539

↓ 1 callersMethodEncode

///////////////////////////////////////////////////////// Simple API.

src/sentencepiece_processor.cc:208

↓ 1 callersMethodEncodeAsSerializedProto

src/sentencepiece_processor.cc:518

↓ 1 callersFunctionExpandUnnormalized

Given an NFKD-normalized string, returns a set of all strings which are normalized into the same `nfkd`. `norm2orig` is the normalized to un-normalize

src/builder.cc:96

↓ 1 callersFunctionFingerprintCat

↓ 1 callersMethodFinish

src/trainer_interface.cc:123

↓ 1 callersMethodGet

src/util.cc:143

↓ 1 callersMethodGetScript

src/unicode_script.cc:28

↓ 1 callersFunctionInitTable

src/unicode_script_map.h:21

↓ 1 callersMethodIsBigram

src/bpe_model_trainer.h:52

↓ 1 callersFunctionIsValidDecodeUTF8

↓ 1 callersFunctionLoadModelProto

src/sentencepiece_processor.cc:704

↓ 1 callersFunctionMakeHeader

src/compile_charsmap_main.cc:100

↓ 1 callersMethodNBestEncode

src/sentencepiece_processor.cc:256

↓ 1 callersMethodNBestEncodeAsSerializedProto

src/sentencepiece_processor.cc:532

↓ 1 callersFunctionRandomString

src/model_interface_test.cc:238

↓ 1 callersMethodReadAll

src/filesystem.cc:55

↓ 1 callersMethodResetVocabulary

src/sentencepiece_processor.cc:161

↓ 1 callersFunctionRunTrainer

src/word_model_trainer_test.cc:35

↓ 1 callersMethodSampleEncodeAsSerializedProto

src/sentencepiece_processor.cc:525

↓ 1 callersMethodSetPrefixMatcher

src/normalizer.h:72

↓ 1 callersFunctionSimpleItoa

↓ 1 callersFunctionToHexData

src/compile_charsmap_main.cc:70

↓ 1 callersFunctionToHexUInt64Array

src/compile_charsmap_main.cc:36

↓ 1 callersFunctionToLogProb

src/unigram_model_trainer.cc:54

↓ 1 callersFunctionToNFC

src/builder.cc:85

↓ 1 callersFunctionToNFKC

src/builder.cc:81

↓ 1 callersFunctionToNFKD

src/builder.cc:77

↓ 1 callersFunctionVerifySpec

src/trainer_interface.cc:52

↓ 1 callersMethodmax_score

Returns the maximum score in sentence pieces. max_score() is used for the cost of user defined symbols.

src/unigram_model.h:136

↓ 1 callersFunctionmix

↓ 1 callersMethodprefix_matcher

src/model_interface.h:68

↓ 1 callersMethodtotal_size

MethodAddNewPair

src/bpe_model_trainer.cc:128

MethodApplyExtraOptions

src/sentencepiece_processor.cc:620

MethodBuildNFKCMap

src/builder.cc:297

MethodBuildNFKC_CFMap

src/builder.cc:455

MethodBuildNmtNFKCMap

src/builder.cc:355

MethodBuildNmtNFKC_CFMap

src/builder.cc:470

MethodBuildTrie

src/unigram_model.cc:435

src/builder.h:38

MethodCompileCharsMap

src/builder.cc:156

MethodComputeFreq

src/bpe_model_trainer.cc:81

Instantiate Trainer instance from trainer_spec and normalization_spec

src/trainer_factory.cc:26

Instantiate Model instance from |model_proto|

src/model_factory.cc:26

MethodDecodeIds

src/sentencepiece_processor.h:279

MethodDecodeIdsAsSerializedProto

src/sentencepiece_processor.cc:546

FunctionDecodePOD

MethodDecodePieces

src/sentencepiece_processor.h:274

MethodDecodePos

Decodes sid, left and right bigram index from uint64.

src/bpe_model_trainer.h:75

MethodDecodePrecompiledCharsMap

src/normalizer.cc:265

MethodDecompileCharsMap

src/builder.cc:223

src/util.cc:153

src/char_model.cc:28

src/word_model.cc:28

src/unigram_model.cc:501

src/sentencepiece_processor_test.cc:470

src/unigram_model_trainer.h:53

MethodEncodeAsIds

src/sentencepiece_processor.h:244

MethodEncodeAsPieces

src/sentencepiece_processor.h:239

FunctionEncodePOD

MethodEncodePos

Encodes sid, left and right bigram index into uint64. Encoded value keeps the order of sid, left and right.

src/bpe_model_trainer.h:65

MethodEncodePrecompiledCharsMap

src/normalizer.cc:254

MethodFinalizeSentencePieces

src/unigram_model_trainer.cc:423

src/freelist.h:28

MethodGetCharSymbol

src/bpe_model_trainer.cc:31

MethodGetNextIndex

src/bpe_model_trainer.cc:112

MethodGetNormalizerSpec

src/sentencepiece_trainer.cc:59

MethodGetPairSymbol

src/bpe_model_trainer.cc:48

MethodGetPieceSize

src/sentencepiece_processor_test.cc:474

MethodGetPieceSize

Returns the size of sentence pieces, which is the same as the size of vocabulary for NMT.

src/model_interface.h:101

← previous next →101–200 of 367, ranked by callers