Return the SymInt placeholder node for the sequence-length dimension of input_ids, or None if it cannot be found.

Flatten the underlying parameters into Arguments: transformer_containers: Iterable of layer containers corresponding to the transfor

x: the activation that we need to quantize num_bits: the number of bits we need to quantize the activation to *args: some ext

Gather partitioned parameters back to full size.

Run generation, return prompt+response in one tensor.

Prepare permutation indices and aligned token counts per expert. Args: tokens_per_expert_group: Token counts for each expert from all ran

Get the data parallel group the caller rank belongs to.

List of ranks in the data parallel group.

Returns an environment variable if it is set and not "", otherwise returns a default value. In contrast, the fallback parameter of os.env

get fp32 gradient partition dictionary accessed as grad_dict[parameter_group_index][parameter_index]

Return preferred granulatity for blocked KV-cache implementation.

converts a log level string into its numerical equivalent. e.g. "info" => `logging.INFO`

Get the pair of lora params and its matched model parameters.

get the associated module name from the model based on the key_word provided by users

Calculates the momentum at batch index. This function treats `self.last_batch_iteration` as the last batch index.

Compute the global norm with MoE experts Inputs: non_expert_norm (float) : the calculated norm of the non-expert params expert_tensors (

Get the existing sequence descriptor for a given uid or initialize one if it does not exist. NOTE: This will always return a valid se

Get the AutoTPConfig object from the configuration. Returns None if no custom config is specified.