MCPcopy Create free account
hub / github.com/davisking/dlib / set_identifier_token

Method set_identifier_token

dlib/tokenizer/tokenizer_kernel_1.cpp:223–244  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

221// ----------------------------------------------------------------------------------------
222
223 void tokenizer_kernel_1::
224 set_identifier_token (
225 const std::string& head_,
226 const std::string& body_
227 )
228 {
229 using namespace std;
230
231 head = head_;
232 body = body_;
233
234 for (unsigned long i = 0; i < UCHAR_MAX; ++i)
235 {
236 headset[i] = false;
237 bodyset[i] = false;
238 }
239
240 for (string::size_type i = 0; i < head.size(); ++i)
241 headset[static_cast<unsigned char>(head[i])] = true;
242 for (string::size_type i = 0; i < body.size(); ++i)
243 bodyset[static_cast<unsigned char>(body[i])] = true;
244 }
245
246// ----------------------------------------------------------------------------------------
247

Callers

nothing calls this directly

Calls 1

sizeMethod · 0.45

Tested by

no test coverage detected