Initialize the weight buffer. The weight buffer is obtained according to the actor. We will construct a large buffer for each dtype i

Compute KL divergence given logprob and ref_logprob. Copied from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1104

adapt from transformers 4.47.1

Load merged state_dict to sharded Megatron module in training.

A memory efficient implementation of logprobs_from_logits

Compute mean of tensor with a masked values.

Whiten values with masked values.

Merge sharded parameters of a Megatron module into a merged checkpoint. Args: wrapped_modelss (list of megatron.model.DistributedDataPara

Create model parameters with meta device. Note buffers in model will still be initialized in default device (e.g., CPU), since the buffe

pad the tokens such that the total length is a multiple of size. This function is useful when applying sequence parallel and context parallel

`Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set.

Generate a function to initialize sub-modules in the `module` with `shard_states` from huggingface checkpoint. Args: module (tor

Parallel load safetensors from huggingface checkpoint Huggingface checkpoint contains: - config.json: a json file for model configurati

Split the data to follow FSDP partition

Remove the pad token. Args: input_ids shape: [bs, seq_length] attention_mask shape: [bs, seq_length] Returns: no_pa

Note that this function only rename the key in the batch

Set input tensor to be used instead of forward()'s input. When doing pipeline parallelism the input from the previous stage comes fro