Args: dataloader: an Iterable of TensorDict that consistently generates prompts. Note that the dataloader should han

A naive rollout. It requires the module to be compatible with huggingface APIs. That is: The module should define __call__ to receive input_id

A vLLM rollout. It requires the module is supported by the vllm. Args: module: module here follows huggingface APIs c

MeagtronPPOActor class. This class implements the simple PPO logics when the model is built with Megatron. Args: config (OmegaCon

The base class for PPO actor Args: config (DictConfig): a config passed to the PPOActor. We expect the type to be

When optimizer is None, it is Reference Policy

LlamaRMSNorm is equivalent to T5LayerNorm

bind the method with actor_prefix to its original name

`List[int]`: List the ids of the special tokens(`'<unk>'`, `'<cls>'`, etc.) mapped to class attributes.

`List[str]`: A list of the unique special tokens (`'<unk>'`, `'<cls>'`, ..., etc.). Convert tokens of `tokenizers.AddedToken` type t

Tensor extenstion to torch.clamp https://github.com/pytorch/pytorch/issues/2793#issuecomment-428784713

Each output must be a DataProto. We concat the dim=0 of output

Only collect output of megatron pp. This is useful when examine weight names as they are identical in tp/dp