Processes WebRED sample and updates relation vocabulary. To process a raw WebRED example, we first extract subj and obj and remove the annotations from the text. The resulting text is parsed with a spacy model to find mention spans, and then tokenized with a BERT tokenizer. If necessary, we
(
sample: Dict[str, Any],
relation_vocab: Dict[str, int],
spacy_model: Any,
tokenizer: Any,
)
| 42 | |
| 43 | |
| 44 | def process_sample( |
| 45 | sample: Dict[str, Any], |
| 46 | relation_vocab: Dict[str, int], |
| 47 | spacy_model: Any, |
| 48 | tokenizer: Any, |
| 49 | ) -> Tuple[Optional[Dict[str, Any]], Dict[str, int]]: |
| 50 | """Processes WebRED sample and updates relation vocabulary. |
| 51 | |
| 52 | To process a raw WebRED example, we first extract subj and obj and remove the |
| 53 | annotations from the text. The resulting text is parsed with a spacy model to |
| 54 | find mention spans, and then tokenized with a BERT tokenizer. If necessary, we |
| 55 | override some spacy mentions with the subj and obj WebRED mentions. |
| 56 | |
| 57 | Args: |
| 58 | sample: raw WebRED sample. Needs to contain following fields: token, list of |
| 59 | token strings. relation, string describing relation between subj and obj. |
| 60 | relation_vocab: dictionary mapping relation strings to integer labels. |
| 61 | spacy_model: spacy model used to detect mentions. |
| 62 | tokenizer: BERT tokenizer. |
| 63 | |
| 64 | Returns: |
| 65 | Processed WebRED sample and updated relation vocabulary. |
| 66 | """ |
| 67 | |
| 68 | processed_sample = {} |
| 69 | |
| 70 | if sample['num_pos_raters'] < 2: |
| 71 | relation = NO_RELATION |
| 72 | else: |
| 73 | relation = sample['relation'] |
| 74 | if relation not in relation_vocab: |
| 75 | relation_vocab[relation] = len(relation_vocab) |
| 76 | label = relation_vocab[relation] |
| 77 | processed_sample['target'] = [label] |
| 78 | |
| 79 | text = sample['annotated_text'] |
| 80 | |
| 81 | # Remove subj and obj annotations from text and store position |
| 82 | def find_span(input_text: str, pattern: Any, |
| 83 | prefix_len: int) -> Tuple[int, int]: |
| 84 | """Find span corresponding to actual subj or obj strings.""" |
| 85 | match = pattern.search(input_text) |
| 86 | span_start = match.start() + prefix_len + 1 |
| 87 | # We want inclusive spans, hence -2 instead of -1 |
| 88 | span_end = match.end() - 2 |
| 89 | return (span_start, span_end) |
| 90 | |
| 91 | def replace_and_adjust( |
| 92 | input_text: str, match: Any, prefix_len: int, |
| 93 | inverted_mapping: np.ndarray) -> Tuple[str, np.ndarray]: |
| 94 | """Remove subj/obj annotations and adjust token mapping accordingly.""" |
| 95 | |
| 96 | original_span_start = match.start() + prefix_len + 1 |
| 97 | original_span_end = match.end() - 1 |
| 98 | actual_string = input_text[original_span_start:original_span_end] |
| 99 | new_text = input_text[:match.start()] + actual_string + input_text[match |
| 100 | .end():] |
| 101 |
no test coverage detected
searching dependent graphs…