MCPcopy Create free account
hub / github.com/catboost/catboost / CutSubtoken

Function CutSubtoken

library/cpp/tokenizer/multitokenutil.cpp:75–85  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

73 }
74
75 inline bool CutSubtoken(TCharSpan& s, size_t maxLen) {
76 if (s.EndPos() > maxLen) {
77 s.Len = maxLen - s.Pos;
78 s.SuffixLen = 0;
79 return true;
80 } else if (s.EndPos() + s.SuffixLen > maxLen) {
81 s.SuffixLen = maxLen - s.EndPos();
82 return true;
83 }
84 return false;
85 }
86}
87
88void CorrectDelimiters(TCharSpan& prevtok, wchar16 suffixChar, TCharSpan& lasttok, wchar16 prefixChar) {

Callers 1

AdjustSubtokensFunction · 0.85

Calls 1

EndPosMethod · 0.80

Tested by

no test coverage detected