Normalize Unicode strings. Necessary for text which contains non-ASCII characters.

Extract the final short answer from the long answer.

Get abstract of each page and hyperplinks within the whole page. :param text: wikipedia text, with hyperlinks

Combine the results from all the chunks. It needs to be implemented by subclasses.

Convert HTML entities entity back to character.

Returns a list of named-entity-recognition tags of each token. Returns None if this annotation was not included.

Generate in-context examples to extract the short answer from the long answer.

Remove the hyperlink from the text. :param text: wikipedia text, with hyperlinks

Return a view of the list of tokens from [i, j).

Returns the original text (with whitespace reinserted).

Returns a list of the text of each token Args: uncased: lower cases text

Args: annotators: None or empty set (only tokenizes).

Args: annotators: None or empty set (only tokenizes).

:param func: The function to run in parallel. :param data: The data to be processed, as an iterable. :param chunk_size: The s

The number of tokens.

Group consecutive entity tokens with the same NER tag.

Returns a list of the lemmatized text of each token. Returns None if this annotation was not included.

Returns a list of all ngrams from length 1 to n. Args: n: upper limit of ngram length uncased: lower cases text

Returns a list of [start, end) character offsets of each token.

Returns a list of part-of-speech tags of each token. Returns None if this annotation was not included.