MCPcopy Create free account
hub / github.com/catboost/catboost / GetTokenCount

Method GetTokenCount

library/cpp/tokenizer/nlpparserbase.h:152–154  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

150 return Tokens[i];
151 }
152 size_t GetTokenCount() const {
153 return Tokens.size();
154 }
155 void BeginToken(const wchar16* tokstart, const wchar16* p) {
156 // it can be called twice in case "exa­́mple", see nlptok.rl, mixedtoken, tokfirst/toknext, numfirst/numnext
157 if (CurCharSpan.Len == 0)

Callers 3

ProcessMultitokenMethod · 0.80
ProcessSurrogatePairsMethod · 0.80
ProcessIdeographsMethod · 0.80

Calls 1

sizeMethod · 0.45

Tested by

no test coverage detected