r""" Configures root logger using a stdout stream handler with an explicit format.

r""" Gets arguments to load config/tokenizer/model. Note: including inplace operation of model_args.

r"""Retrieve topk relevant documents in corpus. Return: list: contains information related to the document, including:

r""" Gets the dict to update the components.

r""" Creates reference model for PPO/DPO training. Evaluation mode is not supported. The valuehead parameter is randomly initialized since it

r""" Finds all available modules to apply LoRA, GaLore or APOLLO.

r""" The model is already unwrapped. There are three cases: 1. full tuning without ds_zero3: state_dict = {"model.layers.*": ..., "v_head

r""" Gets the base elements that are commonly used.

r""" Returns the list of all elements.

r""" Gets logits processor that removes NaN and Inf logits.

r""" Loads model config.

processing query for different encoders

r"""Rerank doc_list.

The overall inference process of a RAG framework.

r""" Gets the response token-by-token of the chat model.

r""" Converts elements to token ids.

r""" Encodes formatted inputs to pairs of token ids. Turn 0: prefix + system + query resp Turn t: sep + query

Compute the log probabilities of the given labels under the given logits. Args: logits: Logits of the model (unnormalized). Shape

r""" Gets the merged datasets in the standard format.

Extracts all n-grams upto a given maximum order from an input segment. Args: segment: text segment from which n-grams will be extracted.

r""" Preprocesses the dataset, including format checking and tokenization.

r""" Computes video sample frames according to fps.

r""" input: a dict with keys {"question", "A", "B", "C", "D", "answer"} output: a tuple of (prompt, response)

r""" Pre-processes a single image.