Create a random mask given input_ids. Support left padding and right padding. Process: - Sample valid token length - Sample left_padding l

exhaustive list of admissible commands

A func to sync attention result with alltoall in sequence parallel gather head dimension and scatter seq dim: e.g. seq_dim: 1, head_dim:

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

Returns list of available actions at the current step

Build the model.

Get a list of all of the *valid* action-object combinations.

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

See: https://github.com/pytorch/pytorch/issues/563#issuecomment-330103591

Lower text and remove punctuation, articles and extra whitespace.

Pad a DataProto to size divisible by size_divisor Args: size_divisor (int): size divisor Returns: data: (DataProto): the pad

Split the batch into a list of micro_batches, where the max_token_len is smaller than max_token_len and the number of valid tokens in each micro b

Updates the conversation history with a new message. Args: role (str): Role of the message sender

broadcast tensor across mp_group

broadcast tensor across mp_group

Returns a string action from poss_acts.

TODO: optimize this. - We can use async ops - We can use only one allgather Args: tensors: size: group:

Replace _flash_attention_forward to _ulysses_flash_attention_forward

TODO: optimize this. Technically, we only need one broadcast

Returns a state representation built from various info sources.

Deprecated. Please use copy_to_local instead.

Write all of the diagnostics from the current iteration level: int. (see logger.py docs) If the global logger level is higher than

The training loop of PPO. The driver process only need to call the compute functions of the worker group through RPC to construct the

end of sentence token can be int or list: 1 or [1, 2] e.g. eos_token=1 response_id: [0, 0, 2, 42, 3, 5, 1, 0, 0] eos_mask: [1, 1,

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap policy

Build the model.