MCPcopy
hub / github.com/BlinkDL/RWKV-LM / get

Method get

RWKV-v5/src/binidx.py:232–245  ·  view source on GitHub ↗

Retrieves a single item from the dataset with the option to only return a portion of the item. get(idx) is the same as [idx] but get() does not support slicing.

(self, idx, offset=0, length=None)

Source from the content-addressed store, hash-verified

230 return sents
231
232 def get(self, idx, offset=0, length=None):
233 """Retrieves a single item from the dataset with the option to only
234 return a portion of the item.
235
236 get(idx) is the same as [idx] but get() does not support slicing.
237 """
238 ptr, size = self._index[idx]
239 if length is None:
240 length = size - offset
241 ptr += offset * np.dtype(self._index.dtype).itemsize
242 np_array = np.frombuffer(
243 self._bin_buffer, dtype=self._index.dtype, count=length, offset=ptr
244 )
245 return np_array
246
247 @property
248 def sizes(self):

Callers 5

run.pyFile · 0.45
__getitem__Method · 0.45
deepspeed_offloadMethod · 0.45
run.pyFile · 0.45
verify.pyFile · 0.45

Calls 1

dtypeMethod · 0.45

Tested by

no test coverage detected