r"""Add the user input to chatbot. Inputs: infer.chatbot, infer.messages, infer.role, infer.query, infer.escape_html Output: infer.ch

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

move the batch to device Args: device (torch.device, str): torch device Returns: DataProto: the curren

Get parallel rank for specified dimension.

Pop a subset of the DataProto via `batch_keys` and `meta_info_keys` Args: batch_keys (list, optional): a list of strings indic

If distributed is initialized, print only on rank 0.

Update the output dict and keep a key in self.output can be used as an attribute.

Split the batch among dim=0 into chunks. The meta_info is passed to each DataProto after split. Args: chunks (int): the number

Get parallel size for specified dimension.

Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up token

Broadcast tensor across specified parallel group.

r"""Optionally check the package version.

r"""Get element by id. Example: top.lang, train.dataset

r"""Check if the environment variable is enabled.

Append a sample to the buffer. Args: item: A sample to append to the buffer. The sample should be a dict with the

Return a tensor with the input `shape` as a view into the 1-D data starting at `start_index`.

r"""Load pretrained tokenizer and optionally loads processor. Note: including inplace operation of model_args.

r"""Register a new composite model. Args: model_type: model type projector_key: multi_modal_projector vision_model_keys:

Barrier all processes.

Copy src from hdfs to local if src is on hdfs or directly return src. If cache_dir is None, we will use the default cache dir of the system. Note

Concat a list of DataProto. The batch is concatenated among dim=0. The meta_info is assumed to be identical and will use the first one.

Load dataset from config.

Check if NPU is available.

r"""Load pretrained model.

A func to sync embedding input with alltoall in sequence parallel gather sequence dimension and scatter head dim: e.g. seq_dim: 1, hea

r"""Get the path to saved model checkpoints.

r"""Automatically select retriever class based on config's retrieval method Args: config (dict): configuration with 'retrieval_method'

Create a huggingface pretrained tokenizer which correctness handles eos and pad tokens. Args: name (str): The name of the tokenizer.

Save the dataset into the original format.

Helper to create a mock sample with a specific token length.

Automatically select generator class based on config.

Select a subset of the DataProto via batch_keys and meta_info_keys Args: batch_keys (list, optional): a list of strings indica

Reduce tensor across specified parallel group.

The evaluation process after finishing overall generation

Repeat the batch data a specified number of times. Args: repeat_times (int): Number of times to repeat the data.

This is the equivalent of torch.repeat_interleave(x, dim=2, repeats=n_rep). The hidden states go from (batch, seqlen, num_key_value_heads,

Create a DataProto from a dict of tensors. This assumes that 1. All the tensor in tensors have the same dim0 2. Only dim0 is the bat

Get ulysses sequence parallel world size.

Get ulysses sequence parallel process group.

pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length. input shape: [bs, seq_length] output shape: [bs, max_se

r"""Save model checkpoint. Subclass and override to inject custom behavior.

r"""Add elements to manager.

Apply the kernel to the model. This method should check if the kernel can be applied (e.g., dependencies are installed, target module

Get current accelerator.

Get process group for specified dimension.

Get parallel local rank.

r"""Plot loss curves and saves the image.

Calculate global rank with support for CP/EP parallelism

r"""Get the number of available devices.

Check if CUDA is available.