Adapted from transformers 4.47.1 to support Ulysses sequence parallelism. NOTE: This function is used for transformers versions in the range

Load merged state_dict to sharded Megatron module in training.

Load merged state_dict to sharded Megatron module in training.

Compute the log_probs from logits with rmpad input_ids and logits. Note that logits_rmpad = model(input_ids_rmpad). For each sentences, there is a

Compute the response log_probs from full logits. Note that logits = model(input_ids) Args: input_ids: [batch_size, seqlen] logits

Compute the log_probs from logits with rmpad logits and pad input. Note that logits_rmpad = model(input_ids_rmpad). For each sentences, there is a

Compute mean of tensor with a masked values.

Whiten values with masked values.

Merge sharded parameters of a Megatron module into a merged checkpoint. Args: wrapped_modelss (list of megatron.core.distributed.Distribu

Merge sharded parameters of a Megatron module into a merged checkpoint. Args: wrapped_models (list of megatron.core.distributed.Distribut

Create model parameters with meta device. Note buffers in model will still be initialized in default device (e.g., CPU), since the buffe

Pad a DataProto to size divisible by size_divisor Args: size_divisor (int): size divisor Returns: data: (DataProto): the pad

pad the tokens such that the total length is a multiple of size. This function is useful when applying sequence parallel and context parallel

`Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set.

Generate a function to initialize sub-modules in the `module` with `shard_states` from huggingface checkpoint. Args: module (tor

Parallel load safetensors from huggingface checkpoint Huggingface checkpoint contains: - config.json: a json file for model configurati

Post-process the output of the model to decode the text. Args: generated_outputs (`torch.Tensor` or `np.ndarray`):

Get chunk data of this tp rank since we do all gather in preprocess.

Split the data to follow FSDP partition

All gather across tp group to make each rank has identical input.

AllGather data from sp region This is because the data is first sharded along the FSDP dimension as we utilize the DP_COMPUTE

Adapted from transformers 4.49.0 to support Ulysses sequence parallelism for transformers >= 4.48.0. NOTE: This function has been tested onl

Adapted from transformers 4.47.1 to support Ulysses sequence parallelism. NOTE: This function is only tested on transformers versions betwee

Remove the pad token. Args: input_ids shape: [bs, seq_length] attention_mask shape: [bs, seq_length] Returns: no_pad

Note that this function only rename the key in the batch

Set input tensor to be used instead of forward()'s input. When doing pipeline parallelism the input from the previous stage comes fro

Set input tensor to be used instead of forward()'s input. When doing pipeline parallelism the input from the previous stage comes fro