Types & classes — github.com/google/sentencepiece

CodeHub MCPcopy Create free account

hub / github.com/google/sentencepiece / types & classes

Types & classes60 in github.com/google/sentencepiece

⨍ Functions 367◇ Types & classes 60

↓ 5 callersClassStatusBuilder

src/deps/status_builder.h:10

↓ 2 callersClassStatus

src/deps/status.h:30

Builder creates a text normalization rule from user-defined string to string mappings. The normalization mapping is compiled into a single and compact

src/builder.h:36

src/deps/status.h:9

ClassDecodeMockModel

src/sentencepiece_processor_test.cc:468

EnumExtraOption

src/sentencepiece_processor.h:361

src/freelist.h:26

ClassGetScriptInternal

src/unicode_script.cc:24

ClassHypothesis

Uses A* search to enumerate N-bests. Given a lattice, enumerates hypotheses (paths) from EOS. At each partial path x, compute f(x) as follows f(x) = g

src/unigram_model.cc:248

ClassHypothesisComparator

src/unigram_model.cc:255

src/util_test.cc:361

Lattice represents a search space of sentence piece segmentation.

src/unigram_model.h:33

src/sentencepiece_processor_test.cc:39

ClassMockPretokenizer

src/pretokenizer_for_training_test.cc:24

Tokenize text into character sequence

src/char_model.h:25

src/unigram_model.h:116

Segmentation model with BPE (Byte Pair Encoding) Details: Neural Machine Translation of Rare Words with Subword Units https://arxiv.org/abs/1508.07909

src/bpe_model.h:30

Tokenize text with whitespaces.

src/word_model.h:25

ClassModelFactory

src/model_factory.h:24

ClassModelInterface

src/sentencepiece_processor.h:85

ClassModelInterface

Underlying model interface. Given a normalized string, returns a sequence of sentence pieces with ids.

src/model_interface.h:46

ClassModelProto

src/sentencepiece_processor.h:87

ClassModelProto

src/model_interface.h:42

ClassNBestSentencePieceText

src/sentencepiece_processor.h:84

src/unigram_model.h:38

ClassNormalizer

src/sentencepiece_processor.h:90

ClassNormalizer

Normalizer implements a simple text normalizer with user-defined string-to-string rules and leftmost longest matching. The rules of Normalizer are bui

src/normalizer.h:64

ClassNormalizerSpec

src/sentencepiece_trainer.h:26

src/bpe_model_trainer.h:57

ClassPosixReadableFile

src/filesystem.cc:32

ClassPosixWritableFile

src/filesystem.cc:70

ClassPrefixMatcher

Given a list of strings, finds the longest string which is a prefix of a query.

src/normalizer.h:36

ClassPretokenizerForTrainingInterface

src/pretokenizer_for_training.h:28

ClassPretokenizerForTrainingInterface

src/sentencepiece_trainer.h:29

ClassRandomGeneratorStorage

src/util.cc:136

ClassReadableFile

src/filesystem.h:29

src/deps/status.h:50

ClassReservoirSampler

src/unicode_script.h:22

ClassSentencePieceProcessor

src/sentencepiece_processor.h:93

ClassSentencePieceText

src/sentencepiece_processor.h:86

ClassSentencePieceTrainer

src/sentencepiece_trainer.h:32

ClassSentenceSelector

src/trainer_interface.cc:101

ClassSpacePattern

Added several corner cases around spaces.

src/normalizer_test.cc:241

ClassStatus::Rep

src/deps/status.cc:10

src/bpe_model.cc:58

src/bpe_model_trainer.h:40

ClassSymbolPair

src/bpe_model.cc:43

ClassSymbolPairComparator

src/bpe_model.cc:50

ClassThreadPool

src/deps/threadpool.h:8

Trainer class for word model. Word model simply counts the frequency of space-delimited tokens, then keep top |vocab_size| frequent tokens.

src/word_model_trainer.h:29

Trainer class for character model.

src/char_model_trainer.h:25

Trainer class for BPE model.

src/bpe_model_trainer.h:30

src/unigram_model_trainer.h:64

ClassTrainerFactory

src/trainer_factory.h:24

ClassTrainerInterface

Base trainer class

src/trainer_interface.h:52

ClassTrainerModel

src/unigram_model_trainer.h:35

ClassTrainerSpec

src/sentencepiece_trainer.h:25

ClassWritableFile

src/filesystem.h:40

Classstring_view_hash