Indicate arrival of warp issuing TMA_STORE

Returns arguments for the transposed problem sizes

Always arrive on even SM of collaborating 2 SMs.

Lightweight update given a subset of arguments

Update output tile index given existing remaining k tiles of current work tile.

Given output tile index, update M, N, L index of current work tile info.

root rank, create multicast obj and export it

non-root rank, import multicast

This function is thread-safe and cached the sm_count. But it will only check the current CUDA device, thus assuming each process handles single GPU.

Helper to rematerialize block Dim. Reduces register liveness.

Helper to rematerialize block Idx. Reduces register liveness.

LUT _e2m1_to_half_x2: Direct E2M1->FP16 (converts 2 E2M1 to 2 FP16)

LUT x8 _e2m1_to_half_x8: Direct E2M1->FP16 (converts 8 E2M1 to 8 FP16 in one call)

Materialize a per-token K-scale tensor [num_batch, num_head_kv, max_kv_pad]. For each request and each KV token position t < kv_seqlens[b]:

Trigger registration side effects for all known modules.

Resolve vLLM symbols used by this backend.

Resolve the CUTLASS MoE API exposed by the selected vLLM checkout.

Pure PyTorch reference matching the new qkpertoken_perhead_vperhead kernel. K processing: per-KV-token scale folded into the FP32 group sum.

Wrap a legacy helper so scalar activation scales are accepted.

PyTorch reference: scatter -inf onto a copy of (logits/T), then argmax.

std::abs

Elementwise add operator (1-by-2)

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Advances an iterator along logical dimensions of matrix in units of whole threadblock tiles

Advances an iterator along logical dimensions of matrix in units of whole warp tiles

Advances an iterator along logical dimensions of matrix in units of whole tiles

Adds a tile offset

Adds a tile offset

Advances an iterator along logical dimensions of matrix in units of whole tiles

Advances an iterator along logical dimensions of matrix in units of whole tiles

Adds a tile offset

Advance shared memory read- and write-iterators to the next stage

Invoke the output functor over each vector of output

Invoke the output functor over each vector of output

Sets the current stream-K work to compute within work_tile_info. If new_unit is true, work_tile_info is populated as a new unit of work. Otherwise, st

Called at the start of the epilogue just before iterating over accumulator slices

Called at the start of a row