MCPcopy Create free account
hub / github.com/AnswerDotAI/ModernBERT / BufferedIterable

Class BufferedIterable

src/sequence_packer.py:479–489  ·  view source on GitHub ↗

Source from the content-addressed store, hash-verified

477
478
479class BufferedIterable(Generic[T]):
480 def __init__(self, iterable: Iterable[T], buffer_size: int):
481 """
482 Args:
483 - iterable: an object which generates a fresh iterator on iter() and which implements len()
484 """
485 self.iterable = iterable
486 self.buffer_size = buffer_size
487
488 def __iter__(self):
489 return BufferedIterator(self.iterable, self.buffer_size)
490
491
492class BufferedIterator(Generic[T]):

Callers 1

build_text_dataloaderFunction · 0.90

Calls

no outgoing calls

Tested by

no test coverage detected