MCPcopy Create free account
hub / github.com/davisking/dlib / tokenize

Method tokenize

examples/text_classification_ex.cpp:204–207  ·  view source on GitHub ↗

The words of a message are the pieces between whitespace and ASCII punctuation.

Source from the content-addressed store, hash-verified

202
203 // The words of a message are the pieces between whitespace and ASCII punctuation.
204 static std::vector<string> tokenize(const string& message)
205 {
206 return split(dlib::tolower(message), " \t\n\r\v\f!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~");
207 }
208
209 static std::pair<sample_type, sample_type> count_ngrams(const std::vector<string>& tokens)
210 {

Callers

nothing calls this directly

Calls 2

tolowerFunction · 0.85
splitFunction · 0.50

Tested by

no test coverage detected