Returns a stride

Returns ImplicitGemm tensor C size in number of elements

Loads a fragment from memory

Loads a fragment

Construct A Single Stage's Accumulator Shape

After reduce call, before smem async fence. Smem stores usually performed here. Upon exit, all smem stores for TMA must have been issued

Issue Tma Descriptor Prefetch -- ideally from a single thread for best performance

Before visit callback. Smem broadcasts usually performed here. Upon entry, all producer loads for this subtile are completed and visible.

Accesses raw internal state

Stores a fragment

After smem async fence, before TMA store commit. Aux stores usually performed here Upon exit, all TMA stores for this subtile must have been issued Be

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element

Adds a tile offset

Advances an iterator along logical dimensions of matrix in units of whole tiles

< Efficiently disables all accesses guarded by mask

Returns whether the current work_tile_info passed in should continue to be used. This occurs only in the stream-K decomposition with stream-K work uni

Returns the biased exponent

Returns a pointer

Return if the address in on the diagonal

Get A extents. fprop: A extents array contains [N,D,H,W,C]. Turn that into ((W,H,D,N), (C)) dgrad: A extents array contains [N,Z,P,Q,K]. Turn that int

Get B extents. fprop: B extents array contains [K,T,R,S,C]. Turn that into ((K), (C,S,R,T)) dgrad: B extents array contains [K,T,R,S,C]. Turn that int

Computes the grid size given a chosen threadblock shape

Determine the number of gemm_k iterations for conv2d problem using implicit gemm algorithm

Loads a fragment from memory at the location pointed to by the iterator.

Loads a fragment from memory with additional logical offset

Loads a fragment from memory with additional logical offset

Loads a fragment from memory

Loads a fragment from memory

Loads a fragment

Obtains raw bits

Python reference that mimics the kernel's numerical flow exactly. Returns (token_ids[B,1], expected penalty_mask after writeback-or-None). I

Changes the size of the view without affecting pointer or layout

Notify the iterator which k-group it is currently pointing to. This does not advance the iterator. Rather, it overrides its internal tracking with co

Store

Store

Store a fragment to memory

Store a fragment to memory

Stores a fragment

/////////////////////////////////////////////////////////////////////////////////////////////// Strided dgrad helper functions

LUT x4 _e2m1_to_bf16_x4: Direct E2M1->BF16 (converts 4 E2M1 to 4 BF16 in one call)

Create random paged FP8 data for dim_qk=128 Stem pipeline.

Adds a pointer offset in units of element

Adds a pointer offset in units of Element

Adds a pointer offset in units of Element