Gets parameters that can be scaled during training. Usually top/last layers. :return: list of parameters

Pad a list of lists to a list of equal length lists. :param inputs: list of lists :param seq_len: seq len to pad to. -1 means take the lo

Initiate wikipedia preprocessing (cleaning of text, removing tables etc.) using wiki_extractor.py Cleaned dump will be around 18GB

Performs an forward pass of ReFinED model and returns the spans for the batch. :param batch: tensors for the batch elements in the ba

Each doc is a list of sentences, each sentence is a list of tokens (tuples) :param file_path: file path :param bio_only: bio

Given a list of dates all of which are from the same doc/page, check to see if can resolve any additional dates by identifying the fo

Used for testing - set attributes not defined in the test instances to None

:param paramsList: the parts of a template or tplarg. Split template parameters at the separator "|". separator "=". Template paramet

Build a dictionary with positional or name key to expanded parameters. :param parameters: the parts[1:] of a template, i.e. all excep

Transform text not containing <nowiki>

Rule based techniques for filtering out any dates which are potentially incorrect

Expands tensors, A, using boolean tensor, B. Where B is a boolean tensor with B.shape = A.shape[:-1]. For example, tensor A could rep

Recursively explores the class hierarchy (parent classes, parent of parents, etc.) Returns all the explored classes (these are all im

Given a list of dates, see if we can identify whether dates in the list are "month_first" or "day_first" format :param dates: list of

Note that this add spans to batch_elements in-place. :param attention_mask: attention mask :param batch_elements: batch_eleme

Removes impliable classes from span (in-place). :param span: span

Split date string into multiple parts, each of which refers to a single date. E.g. "1988 to 1999" -> ["1988", "1999"]. Keeps track of

E2E version of to_batch_elements(). Converts `Doc` to list of batch elements. When the doc has fewer than max sequence length

Adds spans, acc_sum, entity_mask to BatchElement. param spans: list of spans to add :return: no return modifies self in-place

Parser functions have similar syntax as templates, except that the first argument is everything after the first colon. :return: the resul

Check if we can identify whether this dates is in day-first or month-first format

Removes irrelevant parts from :param: text.

Collates a list of tensors for batch elements into a single batch. - Pads tensors to longest length in batch - Corrects masks

Deal with headers, lists, empty sections, residuals of tables. :param text: convert to HTML.

Converts `Doc` into tensors. This method is probably not the method you want to use because it returns tensors for the `Doc` only. Batch

Adds a template defined in the :param page:. @see https://en.wikipedia.org/wiki/Help:Template#Noinclude.2C_includeonly.2C_and_onlyinclude

Concurrently download Wikidata text, Wikipedia redirects, Wikipedia page ids, Wikidata dumps

Additional files which includes files needed to run evaluation on datasets with Wikipedia titles.

Expands template invocation. :param body: the parts of a template. :see http://meta.wikimedia.org/wiki/Help:Expansion for an explanati

r""" Taken from torch code with slight modification to allow Span return value from model forward pass.

Given a surface form (e.g. "Donald Trump") the method will return the top k (MAX_CANDIDATES) based on P(e|m) lookup dictionary. The m

Retrieves all of the classes indices for the qcodes (from various relations used to construct the lookup). :param qcodes: qcodes

Check if the date gives away whether dates in this document are UK style (day/month/year) or US style (month/day/year) :retu

Gets parameters that can be scaled during training. Usually top/last layers. :return: list of parameters