Constructs PreprocessorInferenceOnly object. :param data_dir: path to directory containing the resources files :param max_can

Constructs model for NER and ED. :param config: the config for the model :param preprocessor: preprocessor :param use

Based on transformer's BertForTokenClassification class. Token :param num_labels: number of labels used for token :param drop

:param parameter: the parts of a tplarg.

:param id: id of page. :param title: tutle of page. :param lines: a list of lines.

:param nextFile: a NextFile object from which to obtain filenames to use. :param max_file_size: the maximum size of each

Iterates over the batch elements in each file. Each worker has a different set of files to process. Files contain ~ 1000 batc

Returns number of batch elements. :return: Number of batch elements.

Returns number of batch elements. :return: Number of batch elements.

Initialize the weights.

Initialize the weights.

Initialize the weights.

Initialize the weights.

Initialize the weights.

Initialize the weights.

Add candidate entities to spans in-place. Can return any object to maintain state.

Add candidate entities to spans :param spans: list of spans :param backward_coref: if true will do 2 passes over spans (first

Modifies spans in-place. Checks the predicted fine-grained entity type and type of the predicted entity match. @param self: s

Collates a list of tensors for batch elements into a single batch. - Pads tensors to the longest length in batch - Corrects m

Concurrently download Wikidata text, Wikipedia redirects, Wikipedia page ids, Wikidata dumps

Pull tuples of raw page content, do CPU/regex-heavy fixup, push finished text :param i: process id. :param jobs_queue: where to get jobs.

Post-process options.

Full forward pass including transformer, MD, ET, ED layers. :param batch: BatchedElementsTns is a named tuple containg all the tensor

Forward pass of Mention Detection layer. Treated as token classification problem where each token is assigned a BIO label. B

Forward pass of ED layer. Scores candidate entities. :param class_activations: predicted classes for each mention span. shape

Forward pass of Entity Typing layer. Averages the encoder's contextualised token embeddings for each mention span. :param men

Forward pass :param input_ids: input ids shape = (num_ents, max_cands, 32) 32 = description length, (descriptions) :return:

Load a pretrained model. :param model_file: path to model file :param model_config_file: path to model config file :p

Generates a list of candidate entities (IDs) along with initial/prior scores. The :return: Returns a tuple. First item is a l

Retrieves all of the classes indices for the qcodes (from various relations used to construct the lookup). :param qcodes: qcodes

Gets parameters that can be scaled during training. Usually top/last layers. :return: list of parameters

Gets parameters that can be scaled during training. Usually top/last layers. :return: list of parameters

This implements If_empty from English Wikipedia module: <title>Module:If empty</title> <ns>828</ns> <text>local p = {} f

Set default values for options.

Normalize title

Pull finished article text, write series of files (or stdout) :param opts: global parameters. :param output_queue: text to be output. :par

Convert first arg to roman numeral if <= 5000 else :return: second arg.