MCPcopy Create free account
hub / github.com/DeepRec-AI/DeepRec / TEST

Function TEST

tensorflow/lite/testing/tokenize_test.cc:43–52  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

41}
42
43TEST(TokenizeTest, TokenDetection) {
44 EXPECT_THAT(TokenizeString("x :1"), ElementsAre("x", ":", "1"));
45 EXPECT_THAT(TokenizeString("x:1"), ElementsAre("x", ":", "1"));
46 EXPECT_THAT(TokenizeString("x {1"), ElementsAre("x", "{", "1"));
47 EXPECT_THAT(TokenizeString("x{1"), ElementsAre("x", "{", "1"));
48 EXPECT_THAT(TokenizeString("x }1"), ElementsAre("x", "}", "1"));
49 EXPECT_THAT(TokenizeString("x}1"), ElementsAre("x", "}", "1"));
50 EXPECT_THAT(TokenizeString("x \"1"), ElementsAre("x", "1"));
51 EXPECT_THAT(TokenizeString("x\"1"), ElementsAre("x", "1"));
52}
53
54TEST(TokenizeTest, QuotedTokenDetection) {
55 EXPECT_THAT(TokenizeString("\"w:x{y}z\"1"), ElementsAre("w:x{y}z", "1"));

Callers

nothing calls this directly

Calls 1

TokenizeStringFunction · 0.85

Tested by

no test coverage detected