MCPcopy Create free account
hub / github.com/TIGER-AI-Lab/LongRAG / process_data

Method process_data

utils/mp_util.py:22–27  ·  view source on GitHub ↗
(self)

Source from the content-addressed store, hash-verified

20 self.chunk_size = chunk_size or max(1, len(data) // self.n_processes)
21
22 def process_data(self):
23 data_chunks = [self.data[i:i + self.chunk_size] for i in range(0, len(self.data), self.chunk_size)]
24 with Pool(processes=self.n_processes) as pool:
25 result_chunks = pool.map(self.func, data_chunks)
26 results = self.combine_results(result_chunks)
27 return results
28
29 @staticmethod
30 def combine_results(result_chunks):

Callers 2

Calls 1

combine_resultsMethod · 0.95

Tested by

no test coverage detected