Call the MetaKernel's `apply` to perform the replacement. Corresponding replacement logic is maintained inside each kernel; the only requirem

Replace _flash_attention_forward to _ulysses_flash_attention_forward

TODO: optimize this. Technically, we only need one broadcast

r"""Calculate the number of pixels from the expression.

r"""Get a list of responses of the chat model.

Compresses the given context. Args: context (List[str]): List of context strings that form the basis of the prompt.

Estimate the FLOPS based on the number of valid tokens in the current batch and the time taken. Args: batch_seqlens

Generate sequences

Parse arguments from command line or config file.

r"""Return the list of all elements.

Get FSDP wrap policy for the module. Args: module: The module to get wrap policy for config: Configuration for wrap polic

Get the plugin configuration from the argument value. Args: config: The argument value to get the plugin configuration from. Returns

Get the resource pool of the worker_cls

r"""Initialize the adapters. Support full-parameter, freeze and LoRA training. Note that the trainable parameters must be cast to float32.

Resolve and validate max_tokens parameters from both params and generation_params. Args: params: Dictionary containing user-prov

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

broadcast tensor in tp shards across mp_group

Configure root logger using a stdout stream handler with an explicit format.

r"""Configure root logger using a stdout stream handler with an explicit format.

Convert dataset sample. Args: raw_sample (dict[str, Any]): Raw dataset sample. dataset_name (str): Dataset name.

fetch tensor in tp shards

fetch tensor in tp shards

r"""Get arguments to load config/tokenizer/model. Note: including inplace operation of model_args.

r"""Validate the configuration.

r"""Retrieve topk relevant documents in corpus. Return: list: contains information related to the document, including:

r"""Update gradio components according to the (elem_id, properties) mapping.

Build the memory buffer given weight_buffer_meta Args: weight_buffer_meta: contains mapping from name to a dictionary containing shape

for cuda memory alignment, make sure alignment by 128-bits

r"""Calculate effective tokens per second.

Checks if a given path is vulnerable to LFI. Raises HTTPException if unsafe.

Checks if a given URL is vulnerable to SSRF. Raises HTTPException if unsafe.

Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove su

r"""Works like shutil.copy() for file, and shutil.copytree for dir, and supports hdfs. Copy data and mode bits ("cp src dst"). Return the file'

r"""Create reference model for PPO/DPO training. Evaluation mode is not supported. The valuehead parameter is randomly initialized since it is us

r"""Find all available modules to apply LoRA, GaLore or APOLLO.

r"""Fix proxy settings for gradio ui.

r"""Fix the valuehead checkpoint files. The model is already unwrapped. There are three cases: 1. full tuning without ds_zero3: state_di

r"""Convert dataset examples to messages.

r"""Get the base elements that are commonly used.

r"""Compute the log probabilities of the given labels under the given logits. Returns: logps: A tensor of shape (batch_size,) containing

Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimiz

Get device mesh for specified dimension.