Convert megatron model to transformers model.

This function returns a FusedWorker class. `FusedWorker.{class_name}` -> FusedClass Use `class_name` as a param to directly access t

Create a sampler for the dataset. Arguments: data_config: The data config. dataset (Dataset): The dataset. Returns:

name: name of the parameter param: training parameters infer_params (Iterable[torch.Tensor]): a iterator towards list of para

Args: input_ids: input ids. shape (batch_size, seq_length) attention_mask: attention_mask. shape (batch_size, seq_le

Args: input_ids: input ids. shape (batch_size, seq_length) attention_mask: attention_mask. shape (batch_size, seq_le

We assume: - The model takes input: (input_ids, attention_mask, position_ids). No rmpad for the input - The communication sha

torch.nn.utils.clip_grad_norm_ cann't run on cpu parameter DTensor

Determine the directory name for this rank's checkpoint.

Get the number of gpus in this cluster.

get order of seq lengths to make partitions balanced, this is used in balacing sum of seqlength across dp ranks and microbatches Parameter

Return a dictionary containing name to a shape and dtype.

The answer will be considered correct if: (a) it normalizes to the same string as the ground truth answer OR (b) sympy can simplify t

Check if the given string is an IPv6 address Args: ip_str: The IP address string to check Returns: bool: Returns True i

Extract the last LaTeX boxed expression from a string. Args: string: Input string containing LaTeX code Returns: The last bo

Load merged state_dict to sharded Megatron module in training.

Compute the log_probs from logits with rmpad input_ids and logits. Note that logits_rmpad = model(input_ids_rmpad). For each sentences, there is a

A memory efficient implementation of logprobs_from_logits

Make minibatch iterator for updating the actor Args: data (DataProto): a DataProto containing keys ``input_ids``

Exact match of math if and only if: 1. numerical equal: both can convert to float and are equal 2. symbolic equal: both can convert to sy

Transform the model name in each model_chunk in each pp stage into the name in inference engine

Pad the DataProto by concating with padding_candidate.repeat(padding_size) Args: padding_size (int): the number of repeated paddi

The generation_config created from model config may be different to the pretrained model, this may lead to error when generating: https://git

convert_qkv_gate_up_by_simple_split is a parameter affected by the vLLM version.

AllGather data from sp region This is because the data is first sharded along the FSDP dimension as we utilize the DP_COMPUTE

Release the tool instance. Args: instance_id: The instance id of the tool.

Remove the LaTeX boxed command from a string. Args: s: String with format "\\boxed{content}" Returns: The content inside the

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

if test(generated_code) is not None it'll try to run the code. otherwise it'll just return an input and output pair.

Offload model weights and discard kv cache.

Split the elements in a tuple/interval, while handling well-formatted commas in large numbers

Split the elements in a tuple/interval, while handling well-formatted commas in large numbers