MCPcopy Create free account
hub / github.com/AI-Hypercomputer/maxtext / load_next_batch

Method load_next_batch

src/MaxText/data_loader.py:71–77  ·  view source on GitHub ↗

Loads the next batch with sharding hint

(self, *args, **kwargs)

Source from the content-addressed store, hash-verified

69 return self.last_batch
70
71 def load_next_batch(self, *args, **kwargs):
72 """Loads the next batch with sharding hint"""
73 return maybe_shard_with_name(
74 self.load_next_batch_pre_sharding(),
75 self.input_data_shardings,
76 self.config.shard_mode,
77 )
78
79 def check_example_batch(self):
80 if self.config.max_checkify:

Callers 11

data_load_loopFunction · 0.95
train_loopFunction · 0.95
train_loopFunction · 0.95
train_loopFunction · 0.45
load_next_batchMethod · 0.45
load_next_train_batchMethod · 0.45
generate_completionsFunction · 0.45

Calls 2

maybe_shard_with_nameFunction · 0.90