MCPcopy Create free account
hub / github.com/davisking/dlib / postprocess_decoded_text

Function postprocess_decoded_text

dlib/test/tokenizer.cpp:354–360  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

352 }
353
354 string postprocess_decoded_text(const string& decoded) {
355 string result = decoded;
356 result = regex_replace(result, std::regex("<text>"), "");
357 result = regex_replace(result, std::regex("</text>"), "\n");
358 if (!result.empty() && result.back() == '\n') result.pop_back();
359 return result;
360 }
361
362 template <
363 typename bpe_tok

Callers 1

bpe_tokenizer_testFunction · 0.85

Calls 3

emptyMethod · 0.45
backMethod · 0.45
pop_backMethod · 0.45

Tested by

no test coverage detected