Function clean_mandoc_artifacts

explainshell/extraction/llm/text.py:88–111

Normalize HTML entities mandoc -T markdown emits. ` ` becomes a plain space — its non-breaking semantics aren't needed downstream and ASCII whitespace is friendlier to the LLM. `‌` is preserved. mandoc inserts it between abutting bold/italic emphasis spans (`**foo**‌

(text: str)

def clean_mandoc_artifacts(text: str) -> str:
    """Normalize HTML entities mandoc -T markdown emits.

    ` ` becomes a plain space — its non-breaking semantics aren't needed
    downstream and ASCII whitespace is friendlier to the LLM.

    `‌` is preserved. mandoc inserts it between abutting bold/italic
    emphasis spans (`**foo**‌*bar*`) so CommonMark's delimiter-run
    flanking rules see the `;` (ASCII punctuation) and parse the runs as
    separate spans. Stripping it produced patterns like
    `**--config-file=*****file*` in stored option text, which CommonMark
    folds into garbled emphasis.

    The choice of `‌` is unusual industry-wide — Pandoc and Pod::Markdown
    take different routes (Pandoc emits no separator; Pod::Markdown uses `_`
    for italic to avoid collisions). Both alternatives fail for our corpus:
    Pandoc's "trust CommonMark rule-of-3" misparses the
    `**flag**+arg-name` pattern that mandoc-emitted man pages produce
    constantly; Pod::Markdown's `_` italic breaks intraword italic
    (`_N_th` → literal underscores) which is rampant in real man pages
    (ffmpeg.1, git.1, tar.1, ...). mandoc's `‌` happens to dodge both
    pitfalls for our specific consumer (cmarkgfm + LLM).
    """
    return text.replace(" ", " ")


def filter_sections(text: str) -> tuple[str, dict[str, int]]: