Tracker file rescords the latest chckpoint during training to restart from.

Return a local temp path that joins cache_dir and basename of hdfs_path Args: hdfs_path: cache_dir: Returns:

Get the number of gpus in this cluster.

Note that here we only asks execute_all and execute_rank_zero to be implemented Leave the choice of how these two functions handle argument '

Gets the position ids for Qwen2-VL, it should be generated before sharding the sequence. The batch dim has been removed and the input_ids sho

Get ulysses sequence parallel rank.

Log samples to swanlab as text

Log samples to wandb as a table

Compute the response log_probs from full logits. Note that logits = model(input_ids) Args: input_ids: [batch_size, seqlen] logits

Compute the log_probs from logits with rmpad logits and pad input. Note that logits_rmpad = model(input_ids_rmpad). For each sentences, there is a

with open(wiki_file, 'r') as input: reader = csv.reader(input, delimiter="\t") with open(output_file, 'w') as output:

Main function to run trainer tests. Args: config: Configuration object with training parameters

Compute variance of tensor with masked values.

Transform the model name in each model_chunk in each pp stage into the name in inference engine

offload params of models that are not of current pp rank to cpu

pad a 2D list (e.g. responses, logprobs) to a 2D tensor.

pad the tokens such that the total length is a multiple of sp world size Args: unpad_tokens: (total_nnz, ...). Tokens after removing padd