Flag non-ASCII characters in code, comments, and non-translation strings. Em dashes, smart quotes, arrows, and non-breaking spaces are AI-prose smells that also break older toolchains: MSVC without `/utf-8` mis-decodes them, some legacy editors mojibake them, and grep/diff tools rend
(
lines: list[str], path: Path, fence_mask: list[bool]
)
| 1370 | |
| 1371 | |
| 1372 | def find_non_ascii_violations( |
| 1373 | lines: list[str], path: Path, fence_mask: list[bool] |
| 1374 | ) -> list[Violation]: |
| 1375 | """Flag non-ASCII characters in code, comments, and non-translation |
| 1376 | strings. Em dashes, smart quotes, arrows, and non-breaking spaces are |
| 1377 | AI-prose smells that also break older toolchains: MSVC without `/utf-8` |
| 1378 | mis-decodes them, some legacy editors mojibake them, and grep/diff tools |
| 1379 | render them as escape goo. Words and ASCII operators read fine for both |
| 1380 | humans and LLMs — type `->`, `<=`, `1/2` instead of arrows, less-or-equal |
| 1381 | glyphs, fraction glyphs. |
| 1382 | |
| 1383 | Lines INTENTIONALLY skipped: |
| 1384 | - `// code-verify off` fences, |
| 1385 | - `Copyright ... 20YY-20YY ...` SPDX banner lines (en-dash year range), |
| 1386 | - text inside `qsTr(...)`, `tr(...)`, `QT_TR_NOOP(...)`, etc. — those |
| 1387 | are user-facing localized strings where em dashes, ellipses, ×, °, |
| 1388 | and superscripts are conventional. Only non-ASCII OUTSIDE the |
| 1389 | translation call's argument span is reported.""" |
| 1390 | violations: list[Violation] = [] |
| 1391 | for i, line in enumerate(lines): |
| 1392 | if fence_mask[i]: |
| 1393 | continue |
| 1394 | if _COPYRIGHT_LINE_RE.search(line): |
| 1395 | continue |
| 1396 | |
| 1397 | scan_target = _strip_translation_args(line) if "(" in line else line |
| 1398 | |
| 1399 | offenders: list[str] = [] |
| 1400 | seen: set[str] = set() |
| 1401 | for ch in scan_target: |
| 1402 | if ord(ch) < 128 or ch == "\t": |
| 1403 | continue |
| 1404 | if ch in seen: |
| 1405 | continue |
| 1406 | seen.add(ch) |
| 1407 | offenders.append(_describe_non_ascii(ch)) |
| 1408 | if not offenders: |
| 1409 | continue |
| 1410 | violations.append( |
| 1411 | Violation( |
| 1412 | path, |
| 1413 | i + 1, |
| 1414 | "non-ascii", |
| 1415 | "non-ASCII character(s) — replace with ASCII equivalents: " |
| 1416 | + "; ".join(offenders), |
| 1417 | ) |
| 1418 | ) |
| 1419 | return violations |
| 1420 | |
| 1421 | |
| 1422 | def find_qml_inline_comment_violations( |
no test coverage detected