MCPcopy Create free account
hub / github.com/google-research/language / process_data

Function process_data

language/mentionmemory/data/prepare_complexwq.py:148–164  ·  view source on GitHub ↗

Process CWQ split.

(
    raw_samples: List[Dict[str, Any]],
    spacy_model: Any,
    tokenizer: Any,
)

Source from the content-addressed store, hash-verified

146
147
148def process_data(
149 raw_samples: List[Dict[str, Any]],
150 spacy_model: Any,
151 tokenizer: Any,
152) -> Tuple[List[Dict[str, Any]], int]:
153 """Process CWQ split."""
154
155 processed_samples = []
156 skipped = 0
157 for sample in tqdm.tqdm(raw_samples):
158 processed_sample = process_sample(sample, spacy_model, tokenizer)
159 if processed_sample is None:
160 skipped += 1
161 else:
162 processed_samples.append(processed_sample)
163 logging.info('Skipped %s samples', skipped)
164 return processed_samples, skipped
165
166
167def main(argv: Sequence[str]) -> None:

Callers 1

mainFunction · 0.70

Calls 1

process_sampleFunction · 0.70

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…