Apply NFKC + ligature/smart-quote normalisation + control-char drop. Order matters: NFKC first (so e.g. fullwidth digits collapse to ASCII), then targeted replacements for characters NFKC leaves alone, then drop control characters that PDF extraction commonly leaks.
(text: str)
| 36 | |
| 37 | |
| 38 | def normalize_unicode(text: str) -> str: |
| 39 | """Apply NFKC + ligature/smart-quote normalisation + control-char drop. |
| 40 | |
| 41 | Order matters: NFKC first (so e.g. fullwidth digits collapse to ASCII), |
| 42 | then targeted replacements for characters NFKC leaves alone, then drop |
| 43 | control characters that PDF extraction commonly leaks. |
| 44 | """ |
| 45 | if not text: |
| 46 | return "" |
| 47 | normalized = unicodedata.normalize("NFKC", text) |
| 48 | normalized = _LIGATURE_RE.sub( |
| 49 | lambda m: _LIGATURE_MAP[m.group(0)], normalized |
| 50 | ) |
| 51 | normalized = _CONTROL_RE.sub("", normalized) |
| 52 | return normalized |
| 53 | |
| 54 | |
| 55 | def collapse_whitespace(text: str) -> str: |
no outgoing calls