Get `transformers` library tokenizer. If data directory is passed then look for a folder with the `transformer_name` to and read the tokenize

Initialize weights for all member variables with type nn.Module

Initialize weights for all member variables with type nn.Module

Initialize weights for all member variables with type nn.Module

Initialize weights for all member variables with type nn.Module

Initialize weights for all member variables with type nn.Module

Initialize weights for all member variables with type nn.Module

:return: a string with its first character lowercase

After the package is unpacked at the target destination, the requirements can be installed locally from the wheelhouse folder using t

Function applied to wikiLinks

Merges main entity mentions into spans (hyperlinks_clean). :param title: Wikipedia title. :param spans: spans (hyperlinks) wi

:param input_file: name of the wikipedia dump file; '-' to read from stdin :param template_file: optional file with template definitions.

Prune classes that are inferred from other provided classes. Note that this also filters country and sport relation as well.

Create a new version of a dataset using MentionDetector to add additional NER labels (e.g. for DATE spans)

Add new NER labels to a doc, using a trained MentionDetector

https://www.mediawiki.org/wiki/Help:Links#External_links [URL anchor text]

Replaces internal links of the form: [[title |...|label]]trail with title concatenated with trail, when present, e.g. 's' for plural.

Resolve a single date -> i.e. convert raw text form to an instance of the Date class :param span: the span to resolve to a date

Given a list of spans with MD type "DATE", attempt to resolve them to a set date format. The list of spans should all be from the sam

Based on https://discuss.pytorch.org/t/how-to-sort-tensor-by-given-order/61625. :param data_tns: input tensor :param permutat

Given a text, generates (start, end) spans of sentences in the text. @param text: input text @return: list of tuple o

convert integer to Roman numeral

Convert `BatchElement` to labels for NER. param ner_tag_to_ix: dict mapping from ner label (e.g. "B-PERSON") to index in output layer

Attempt to convert date to a string

:param out: a memory file :param text: the text of the page

Writes memmap-based data structure to disk in compacted format and deletes original over-allocated file. Only call this metho

Returns the tensors for a single batch element. :param index: the index of the batch element to return :return: batch element

Constructs instance of Refined class. :param data_dir: directory containing the data files (pickled and JSON dictionaries) :p

Constructs instance of Wikipedia dataset (should be used in conjunction with torch Dataloader). :param preprocessor: preprocessor pro

Constructs dataset object. :param preprocessor: utility class that provides methods for Wikidata lookups and general pre-processing

Constructs dataset object. :param preprocessor: utility class that provides methods for WikiData lookups and general pre-processing

:param boto3_session: boto3 session (`boto3.Session`) optional, if not set will use default session High-level abstractions f