MCPcopy Create free account
hub / github.com/THUDM/GLM / data_iterator

Method data_iterator

data_utils/samplers.py:152–160  ·  view source on GitHub ↗

iterates through data and handles wrap around

(self, _iter, wrap_around=False)

Source from the content-addressed store, hash-verified

150 self.sampler.wrap_around += self.batch_size
151
152 def data_iterator(self, _iter, wrap_around=False):
153 """iterates through data and handles wrap around"""
154 for i, idx in enumerate(_iter):
155 if i < self.wrap_around%self.batch_size:
156 continue
157 if wrap_around:
158 self.wrap_around += 1
159 self.wrap_around %= self.batch_size
160 yield idx
161
162 def _batch(self, batch):
163 """extracts samples only pertaining to this worker's batch"""

Callers 1

__iter__Method · 0.95

Calls

no outgoing calls

Tested by

no test coverage detected