Function unescape_html

ftfy/fixes.py:94–139

Decode HTML entities and character references, including some nonstandard ones written in all-caps. Python has a built-in called `html.unescape` that can decode HTML escapes, including a bunch of messy edge cases such as decoding escapes without semicolons such as "&amp".

(text: str)

def unescape_html(text: str) -> str:
    """
    Decode HTML entities and character references, including some nonstandard
    ones written in all-caps.

    Python has a built-in called `html.unescape` that can decode HTML escapes,
    including a bunch of messy edge cases such as decoding escapes without
    semicolons such as "&amp".

    If you know you've got HTML-escaped text, applying `html.unescape` is the
    right way to convert it to plain text. But in ambiguous situations, that
    would create false positives. For example, the informally written text
    "this&not that" should not automatically be decoded as "this¬ that".

    In this function, we decode the escape sequences that appear in the
    `html.entities.html5` dictionary, as long as they are the unambiguous ones
    that end in semicolons.

    We also decode all-caps versions of Latin letters and common symbols.
    If a database contains the name 'PÉREZ', we can read that and intuit
    that it was supposed to say 'PÉREZ'. This is limited to a smaller set of
    entities, because there are many instances where entity names are
    case-sensitive in complicated ways.

        >>> unescape_html('<tag>')
        '<tag>'

        >>> unescape_html('&Jscr;ohn &HilbertSpace;ancock')
        '𝒥ohn ℋancock'

        >>> unescape_html('&checkmark;')
        '✓'

        >>> unescape_html('P&eacute;rez')
        'Pérez'

        >>> unescape_html('P&EACUTE;REZ')
        'PÉREZ'

        >>> unescape_html('BUNDESSTRA&SZLIG;E')
        'BUNDESSTRASSE'

        >>> unescape_html('&ntilde; &Ntilde; &NTILDE; &nTILDE;')
        'ñ Ñ Ñ &nTILDE;'
    """
    return HTML_ENTITY_RE.sub(_unescape_fixup, text)


ANSI_RE = re.compile("\033\\[((?:\\d|;)*)([a-zA-Z])")