MCPcopy Create free account
hub / github.com/google-research/language / process_sample

Function process_sample

language/mentionmemory/data/prepare_webred.py:44–242  ·  view source on GitHub ↗

Processes WebRED sample and updates relation vocabulary. To process a raw WebRED example, we first extract subj and obj and remove the annotations from the text. The resulting text is parsed with a spacy model to find mention spans, and then tokenized with a BERT tokenizer. If necessary, we

(
    sample: Dict[str, Any],
    relation_vocab: Dict[str, int],
    spacy_model: Any,
    tokenizer: Any,
)

Source from the content-addressed store, hash-verified

42
43
44def process_sample(
45 sample: Dict[str, Any],
46 relation_vocab: Dict[str, int],
47 spacy_model: Any,
48 tokenizer: Any,
49) -> Tuple[Optional[Dict[str, Any]], Dict[str, int]]:
50 """Processes WebRED sample and updates relation vocabulary.
51
52 To process a raw WebRED example, we first extract subj and obj and remove the
53 annotations from the text. The resulting text is parsed with a spacy model to
54 find mention spans, and then tokenized with a BERT tokenizer. If necessary, we
55 override some spacy mentions with the subj and obj WebRED mentions.
56
57 Args:
58 sample: raw WebRED sample. Needs to contain following fields: token, list of
59 token strings. relation, string describing relation between subj and obj.
60 relation_vocab: dictionary mapping relation strings to integer labels.
61 spacy_model: spacy model used to detect mentions.
62 tokenizer: BERT tokenizer.
63
64 Returns:
65 Processed WebRED sample and updated relation vocabulary.
66 """
67
68 processed_sample = {}
69
70 if sample['num_pos_raters'] < 2:
71 relation = NO_RELATION
72 else:
73 relation = sample['relation']
74 if relation not in relation_vocab:
75 relation_vocab[relation] = len(relation_vocab)
76 label = relation_vocab[relation]
77 processed_sample['target'] = [label]
78
79 text = sample['annotated_text']
80
81 # Remove subj and obj annotations from text and store position
82 def find_span(input_text: str, pattern: Any,
83 prefix_len: int) -> Tuple[int, int]:
84 """Find span corresponding to actual subj or obj strings."""
85 match = pattern.search(input_text)
86 span_start = match.start() + prefix_len + 1
87 # We want inclusive spans, hence -2 instead of -1
88 span_end = match.end() - 2
89 return (span_start, span_end)
90
91 def replace_and_adjust(
92 input_text: str, match: Any, prefix_len: int,
93 inverted_mapping: np.ndarray) -> Tuple[str, np.ndarray]:
94 """Remove subj/obj annotations and adjust token mapping accordingly."""
95
96 original_span_start = match.start() + prefix_len + 1
97 original_span_end = match.end() - 1
98 actual_string = input_text[original_span_start:original_span_end]
99 new_text = input_text[:match.start()] + actual_string + input_text[match
100 .end():]
101

Callers 1

process_dataFunction · 0.70

Calls 4

find_spanFunction · 0.85
replace_and_adjustFunction · 0.85
padMethod · 0.80
overlapsFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…