Recursively compares two NumPy arrays for strict equality, with special handling for object-dtype arrays, NaN values, and circular references

Reorder the data on single controller such that each dp rank gets similar total tokens

Binds the methods of each worker to the WorkerDict. Note that we only bind public methods that are decorated by register

Blocks until all workers in the group are alive.

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

Build prompt for batch generation.

Build prompt for batch generation.

Build prompt for batch generation.

Build prompt for image-based QA generation. Args: task_instruction: Task description input_instruction: Input format

Replace <image> and <video> placeholder in messages with corresponding image and video which is required by processor.apply_chat_template.

Build prompt for sample generation (extracted from generate_sample for batching). Args: task_instruction: The task descr

Calculates the total number of shards and the shape of the device mesh.

Call tool and return tool response.

Checks if the key is a valid Megatron state key. Now the model merger only supports keys that start with "decoder/embedding/output_l

Validate mutually exclusive micro batch size configuration options. Ensures that users don't set both deprecated micro_batch_size and

Check if the resource pool can be satisfied in this ray cluster.

Forward method with activation checkpointing.

Remove image references, table tags, and clean up markdown for text retrieval.

Release vLLM, SentenceTransformer, and Translation models, free GPU memory after evaluation completes.

Clear all buffer directories after successful pipeline completion.

Clear the checkpoint directory to avoid reloading old checkpoints.

Clear generation phase buffers after successful completion.

Clear refinement phase buffers after successful completion.

Clear the training upload directory.

Clear the upload directory.

This function should only be called inside by WorkerGroup

Convert config RubricItem list to DeepEval Rubric list.

Convert config RubricItem list to DeepEval Rubric list.

Creates the train and validation dataloaders.

Patches flash attention forward to handle 3D position ids in mrope. (3, batch_size, seq_length)

Patches flash attention forward to handle 3D position ids in mrope. (3, batch_size, seq_length)

Detect the type of image data and validate it can be retrieved. Args: image_data: The image data from the dataset sample

- megatron: base class should be MegatronWorker - fsdp: base class should be Worker

Determine the path to resume from based on resume_mode configuration

Determine the path to resume from based on resume_mode configuration

hdfs capable to check whether a file_path is exists

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.