MCPcopy Create free account
hub / github.com/google/sentencepiece / SampleEncode

Method SampleEncode

src/unigram_model.cc:544–560  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

542}
543
544EncodeResult Model::SampleEncode(absl::string_view normalized,
545 float theta) const {
546 if (!status().ok() || normalized.empty()) {
547 return {};
548 }
549
550 Lattice lattice;
551 lattice.SetSentence(normalized);
552 PopulateNodes(&lattice);
553
554 EncodeResult results;
555 for (const auto *node : lattice.Sample(theta)) {
556 results.emplace_back(node->piece, node->id);
557 }
558
559 return results;
560}
561
562} // namespace unigram
563} // namespace sentencepiece

Callers 5

TESTFunction · 0.45
TESTFunction · 0.45
mainFunction · 0.45
TESTFunction · 0.45
TESTFunction · 0.45

Calls 3

okMethod · 0.80
SetSentenceMethod · 0.80
SampleMethod · 0.80

Tested by 4

TESTFunction · 0.36
TESTFunction · 0.36
TESTFunction · 0.36
TESTFunction · 0.36