Load a pretrained ReFinED model. :param model_name: model name (e.g. wikipedia_model) :param data_dir: data directory path (b

:param wikitext: the text to be expanded. Templates are frequently nested. Occasionally, parsing mistakes may cause template

Predicts the types and entity for each span (if spans are not provided they will be identified). Performs the preprocessing and batch

Batches an iterables by yielding lists of length n. Final batch length may be less than n. :param iterable: any iterable :param n: batch

:return: a string with just its first character uppercase We can't use title() since it coverts all words.

Build a dictionary of var/value from :param: args. Parameters can be either named or unnamed. In the latter case, their name is taken fro

Collate function for mention (span) detection :param pad_id: value to use for padding :return: collate function (can be unsed in pytorch

Download a file from S3 and store at output_file_path if it has not already been downloaded. Only downloads file if a more recent tim

Construct `Doc` from text with predetermined spans. :param text: document text :param spans: document spans (with document ch

Converts `Doc` to list of batch elements. When the doc has fewer than max sequence length tokens then the return list is a singleton

Transforms wiki markup. @see https://www.mediawiki.org/wiki/Help:Formatting

From a set of (direct) classes this method will generate all of the classes that can be implied. When remove_self is True it means th

Converts batch element to tuple of tensors. :param batch_element: batch element to convert :param processor: preprocessor used to lookup

:param batch_elements: elements to batch. :param preprocessor: doc preprocessor. :param max_batch_size: max batch size. :param sort_b

A matching function for nested expressions, e.g. namespaces and tables.

This method is used for end-to-end entity linking from text :param text: text :param preprocessor: preprocessor :retu

Retrieves all of the classes indices for the qcodes (from various relations used to construct the lookup). :param qcodes: qcodes

In-place sort spans :param spans: spans

Removes HTML or XML character references and entities from a text string. :param text The HTML (or XML) source text. :return The plain te

Adds candidate entities to each span. :param spans: list of spans :param backward_coref: if true will do 2 passes over spans

Collate (pad, correct index mask, stack) a list of tuples of tensors produced from BatchElement.to_tensors(). Pads tensors to longest batch e

:param ldelim: number of braces to match. 0 means match [[]], {{}} and {{{}}}.

Load templates from :param file:. :param output_file: file where to save templates and modules.

Masks entity mentions in the batch element a random percentage of the time with the [MASK] token and sometimes with a random token from the v

Merge to lists of spans. It ensures the spans do not overlap. When there is overlap prioritised_spans is picked. :param additional_spans: the

Scans input extracting pages. :return: (id, revid, title, namespace key, page), page is a list of lines.

Batched version of process_text(). This method uses the convert_batch_elements_to_batched_tns() method which could be much more

Given a text input representing a date, resolve it to a list of Date instances :param text: the text representing the date :r

Given a segment of text representing a date, try and resolve it to a Date instance e.g. "20/01/2020" -> Date(day=20, month=1, year=20

Removes impliable classes from spans (in-place). :param spans: spans

Applies some minor corrections to a list of spans in-place. An example is removing newline characters from start and end of end. :param s

Additional files which includes files needed to run evaluation on datasets with Wikipedia titles.

Drop from text the blocks identified in :param spans:, possibly nested.

The execute command will loop and keep on reading the stdout and check for the return code and displays the output in real time.

:param out: a memory file.

Creates a tensor of shape (bs, num_spans, contexutalised_embedding_size). The tns argument is the contextualised token embeddings is grouped

Assuming that text contains a properly balanced expression using :param openDelim: as opening delimiters and :param closeDelim: as closin

Determine the namespace of the page being included through the template mechanism

Generates a list of candidate entities for a surface form using a pem lookup. Person name coreference is done by copying pem values f

Retrieves descriptions input_ids for a batch of qcodes and optionally reshapes tensor. :param qcodes: qcodes :param shape: sh

Retrieves descriptions input_ids for a batch of qcodes and optionally reshapes tensor. :param qcodes: qcodes :param shape: sh