Synchronous wrapper for `self.async_step`.

Sends a signal to gracefully stop all inference engines. The signal first propagates thru the coordinator to all engines. All engines

Remove ANSI escape codes (terminal formatting) from text. Griffe includes ANSI codes for terminal formatting in some strings, which break

Performs SwiGLU (Swish-Gated Linear Unit) activation function. Args: y (torch.Tensor): Input tensor to be split into two halves along the

Converts text to tokens.

Returns the local shard of the given tensor if it is a DTensor.

Encoding function.

Returns id of unknown tokens.

Update the output JSON file with results from a single instance.

Update the DTensor's chunk metadata to handle uneven sharding. This function modifies the DTensor in-place to include chunk metadata and

Vocab range from global vocab size.

Waits for the event to complete on the current CUDA stream.

Wait for the previous reduce-scatter/all-reduce to finish. Args: suggested_queue_size (int, optional): The recommended qu

Creates a weak reference to a method to prevent circular references. This function creates a weak reference to a method and returns a wrapper fun

Creates metadata file for tokenizer. Args: tokenizer_path (str): path to tokenizer model. tokenizer_library

Initialize the DeepEP dispatcher. Args: group (torch.distributed.ProcessGroup): The process group to use for communicati

Receive happens between the layers with size [seq_len, batch_size, hidden_size].

Best-effort ``docker update --memory`` on a running container.

Convert config objects to dicts.

Initialize tokenizer.

Forward a chat completions request to the Fireworks deployment. On HTTP 425 (model hot-loading), automatically extends the retry budget

Casts weights to the expected datatype. When appropriate scaling factor is found inside self.scales, the weight gets scaled before the cas

Get value from src_dict according to the key, and copy value to dist_dict with new key. The new key is generated by formatting old ke

This function is only used for MXFP8 params. Copy FP32 main params directly to param buffer for param all-gather since param

Get instance details

Detokenize a sequence of token IDs, handling skip_special_tokens for different tokenizer APIs. On the first call, inspects `

Split shared embedding parameters into separate bucket if using distributed optimizer that makes use of reduce-scatters instead of al

Calculate top-n log probs from logits for dynamic batching. Args: logits (Tensor): The logits to compute top-n log probs from.

Perform bookkeeping necessary to sample logits for dynamic batching. The ability to override the context's data is solely intended for

Validate and evaluate a string containing a Python list expression

If self.fim_split_sample is not None, split the sequence. Then apply FIM on the fragments, or the whole sequence if self.fim_split_sa

Find sub-module in model

Extract plain text from multimodal content lists.

Format proxy URL

Runs a forward pass of the model. Args: inference_input(Dict[str, Any]): The input data. Returns: The model

Forward inference request to Tinker SamplingClient.

Get all the rng states.

Returns the batch size and sequence length based on the tokens tensor and recv_buffer_seq_len. Args: tokens (torch.Tenso

Get the full class name with module path.

Return the random number generator state of the specified GPU. Arguments: device (int): The gpu to retrieve the rng state clone (

Build the data parallel buffer shard bucket index from the bucket index. Args: bucket_index (BucketIndex): The bucket index containi

Get current energy consumption from NVML.

Return a dict containing the main param and optimizer states corresponding to the input model_param. The structure of the returned di

Get Megatron optimizer based on parameter groups. Args: config (OptimizerConfig): optimizer configuration object. model_chunks (l

Returns parameter groups and buffer for optimizer. Args: model_chunks (List[MegatronModule]): model chunks to create parameter

Load the currently selected trajectory file.

Maintain the expert bias in float32. When using bf16/fp16, the expert bias gets converted to lower precision in Float16Module.

Submit turns whose PRM is done (or PRM not needed). force_no_prm: also submit turns that have no PRM task yet (used at session end fo

Normalize messages for the chat template (developer -> system, flatten content).

verl-aligned top-K OPD surrogate for one sample. Pipeline (notation matches the file docstring): 1. Build ``S_t = student_indices ∩ teach