Abstract class representing a preprocessor that provides methods to use resources such as Wikidata classes, and P(e|m) alias table. Implementations could memory-based, disk-based, or network-based depending on memory requirements.
| 20 | |
| 21 | @dataclass |
| 22 | class Preprocessor(ABC): |
| 23 | """ |
| 24 | Abstract class representing a preprocessor that provides methods to use resources such as Wikidata classes, and |
| 25 | P(e|m) alias table. Implementations could memory-based, disk-based, or network-based depending on |
| 26 | memory requirements. |
| 27 | """ |
| 28 | |
| 29 | data_dir: str |
| 30 | transformer_name: str |
| 31 | max_candidates: int |
| 32 | ner_tag_to_ix: Dict[str, int] |
| 33 | num_classes: int = field(init=False) |
| 34 | max_num_classes_per_ent: int = field(init=False) |
| 35 | cls_id: int = field(init=False) |
| 36 | sep_id: int = field(init=False) |
| 37 | pad_id: int = field(init=False) |
| 38 | tokenizer: PreTrainedTokenizer = field(init=False) |
| 39 | transformer_config: PretrainedConfig = field(init=False) |
| 40 | transformer_model: PreTrainedModel = field(init=False) |
| 41 | max_seq: int = 510 |
| 42 | |
| 43 | @abstractmethod |
| 44 | def get_classes_idx_for_qcode_batch( |
| 45 | self, qcodes: List[str], shape: Tuple = None |
| 46 | ) -> torch.LongTensor: |
| 47 | """ |
| 48 | Retrieves all of the classes indices for the qcodes (from various relations used to construct the lookup). |
| 49 | :param qcodes: qcodes |
| 50 | :param shape: shape/size of tensor returned (int) |
| 51 | :return: long tensor with shape (num_ents, self.max_num_classes) (qcode index 0 is used for padding all classes) |
| 52 | """ |
| 53 | pass |
| 54 | |
| 55 | @abstractmethod |
| 56 | def get_candidates( |
| 57 | self, |
| 58 | surface_form: str, |
| 59 | person_coref_ref: Dict[str, List[Tuple[str, float]]] = None, |
| 60 | sample_k_candidates: Optional[int] = None |
| 61 | ) -> Tuple[List[Tuple[str, float]], Dict[str, Any]]: |
| 62 | """ |
| 63 | Given a surface form (e.g. "Donald Trump") the method will return the top k (MAX_CANDIDATES) based on |
| 64 | P(e|m) lookup dictionary. The method returns List[(qcodes, pem_value, pme_value)], person_coref. |
| 65 | Person coref dictionary keeps track of humans mentioned in the document and copies the values from full name |
| 66 | over partial name mentions (e.g. from "Donald Trump" to "Trump"). |
| 67 | :param surface_form: surface form to fetch candidates for |
| 68 | :param person_coref_ref: a dictionary of previous human mentions with partial names in the dictionary |
| 69 | :param sample_k_candidates: randomly sample candidates (hard, random, and gold) |
| 70 | :return: List[(qcode, pem_value)], person_coref |
| 71 | """ |
| 72 | pass |
| 73 | |
| 74 | @abstractmethod |
| 75 | def add_candidates_to_spans( |
| 76 | self, |
| 77 | spans: List['Span'], |
| 78 | backward_coref: bool = False, |
| 79 | person_coreference: Optional[Dict[str, List[Tuple[str, float]]]] = None, |
nothing calls this directly
no outgoing calls
no test coverage detected