MCPcopy Create free account
hub / github.com/google-research/language / process_data

Function process_data

language/mentionmemory/data/prepare_fever.py:54–125  ·  view source on GitHub ↗

Processes fever splits. Given claims and accompanying labels, tokenizes the claims and annotates claim with mention positions corresponding to noun chunks. Args: data: List of raw FEVER samples. Each raw FEVER sample contains: claim - text that describes the claim to be classified.

(data: Sequence[Dict[Text, Any]], spacy_model: Any,
                 tokenizer: Any)

Source from the content-addressed store, hash-verified

52
53
54def process_data(data: Sequence[Dict[Text, Any]], spacy_model: Any,
55 tokenizer: Any):
56 """Processes fever splits.
57
58 Given claims and accompanying labels, tokenizes the claims and annotates
59 claim with mention positions corresponding to noun chunks.
60
61 Args:
62 data: List of raw FEVER samples. Each raw FEVER sample contains: claim -
63 text that describes the claim to be classified. label - string label,
64 REFUTES, SUPPORTS or NOT ENOUGH INFO.
65 spacy_model: Spacy model object.
66 tokenizer: BERT tokenizer.
67
68 Returns:
69 List of processed FEVER samples. Each sample contains:
70 text_ids: dense token ids.
71 text_mask: dense padding mask.
72 mention_start_positions: sparse array of mention starting positions.
73 mention_end_positions: sparse array of mention end positions.
74 mention_mask: mention padding mask.
75 target: string labels converted to 0, 1, 2.
76 """
77 processed_data = []
78 for sample in tqdm.tqdm(data):
79 new_sample = {}
80 parsed_claim = spacy_model(sample['claim'])
81 mention_char_spans = []
82 for chunk in parsed_claim.noun_chunks:
83 span_start_char = parsed_claim[chunk.start].idx
84 span_last_token = parsed_claim[chunk.end - 1]
85 span_end_char = span_last_token.idx + len(span_last_token.text) - 1
86 mention_char_spans.append((span_start_char, span_end_char))
87
88 _, text_ids, text_mask, mention_spans, _ = tokenization_utils.tokenize_with_mention_spans(
89 tokenizer=tokenizer,
90 sentence=sample['claim'],
91 spans=mention_char_spans,
92 max_length=FLAGS.max_length,
93 add_bert_tokens=True,
94 )
95
96 new_sample['text_ids'] = text_ids
97 new_sample['text_mask'] = text_mask
98
99 mention_spans = np.array(mention_spans)
100 if len(mention_spans) == 0: # pylint: disable=g-explicit-length-test
101 mention_start_positions = np.array([], dtype=np.int64)
102 mention_end_positions = np.array([], dtype=np.int64)
103 else:
104 mention_start_positions = mention_spans[:, 0]
105 mention_end_positions = mention_spans[:, 1]
106
107 mention_start_positions = mention_start_positions[:FLAGS.max_mentions]
108 mention_end_positions = mention_end_positions[:FLAGS.max_mentions]
109
110 mention_pad_shape = (0, FLAGS.max_mentions - len(mention_start_positions))
111

Callers 1

mainFunction · 0.70

Calls 1

padMethod · 0.80

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…