Get the size of the Ulysses sequence parallel group.

Get the rank of this process in the Ulysses sequence parallel group the caller rank belongs to.

Activates gradient checkpointing for the current model. Note that in other frameworks this feature can be referred to as "activation

Compute a hash of the input string and return the first 16 characters of the hash.

Check if the model at the given path is a visual language model. Args: model_path (str): The path to the model. Returns:

Checks to see whether the pixel values have already been rescaled to [0, 1].

Iterate over the ranges in a random order.

Whiten values with masked values.

Context manager to globally disable weight initialization to speed up loading large models. TODO(Patrick): Delete safety argument `_enable=T

Rebase the URLs in a shardlist.

This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, he

Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. Takes care of tying weights embeddings

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Rotates half the hidden dims of the input.

Saves all of the model weights

Predict the sample at the previous timestep by reversing the SDE. Core function to propagate the diffusion process from the learned model ou

Sets the progress bar config for the pipeline

Print the colour legend.

Extract frames from a video and save them to the output directory. Parameters: video_file_name (str): Path to the video file. num_fr

Resize tokenizer and embedding. Note: This is the unoptimized version that may make your embedding size not be divisible by 64.

Returns the basename and extension of a filename

r""" Returns the state dictionary of the model. We add the state dictionary of the value head to the state dictionary of the wrapped m

Perform a single step of training. Args: epoch (int): The current epoch. global_step (int): The current glob

Check if the current generation sequence has finished.

Check if the current generation sequences have finished.

Returns the text encoder used for encoding text inputs

:param pred: :param truth: :param alpha: :return:

Calculates the running mean and standard deviation of a data stream. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f

Checks the availability of Flash Attention 2 and compatibility with the current model. If all checks pass and `hard_check_only` is F

Checks the availability of SDPA for a given model. If all checks pass and `hard_check_only` is False, the method will set the config

Create a clone of the world group Note: We need to clone the dist world group because we use dist.get_global_rank() utility function in DeepSp

compute the accuracy (soft score) of human answers

Converts a BLOOM Causal LM to a Prefix LM. Supported HuggingFace model classes: - `BloomForCausalLM` See `convert_hf_causal_lm_to_pr

Converts a GPT-style Causal LM to a Prefix LM. Supported HuggingFace model classes: - `GPT2LMHeadModel` - `GPTNeoForCausalLM`

-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]

Converts an OPT Causal LM to a Prefix LM. Supported HuggingFace model classes: - `OPTForCausalLM` See `convert_hf_causal_lm_to_prefi

r""" Handles the early stopping logic. If the policy KL is greater than the target KL, then the gradient is zeroed and the optimizatio