MCPcopy Create free account
hub / github.com/alibaba/MNN / cs_punct

Function cs_punct

transformers/llm/engine/src/tokenizer/unicode.cpp:143–149  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

141}
142static CharSet cs_mark() { CharSet cs; cs.set(0x05); return cs; }
143static CharSet cs_punct() {
144 CharSet cs;
145 cs.set(0x06);
146 const char* p = "!\"#%&'()*,-./:;?@[\\]_{}";
147 while (*p) cs.set((uint8_t)*p++);
148 return cs;
149}
150static CharSet cs_symbol() {
151 CharSet cs;
152 cs.set(0x07);

Callers 1

prop_charsetMethod · 0.85

Calls 1

setMethod · 0.45

Tested by

no test coverage detected