Helper to rematerialize block Idx. Reduces register liveness.

Loads a fragment from memory with additional logical offset

Loads a fragment from memory with additional logical offset

Issue Tma Descriptor Prefetch -- ideally from a single thread for best performance

Stores a fragment to memory with additional pointer offset

Computes the quotient and remainder in a single method.

* Find divisor, using find_log2 */

Initializes members. This variant of the method should only be used when problem_shape and tile_shape contain modes of only rank 1.

Helper to rematerialize block Idx. Reduces register liveness.

* Performs a non-blocking allocation of TMEM. * @param num_columns Number of columns being freed. Must be 32 <= num_columns <= 512 and power of 2.

Returns a warp-uniform value indicating the canonical warp group index of the calling threads. Threads within the warp must be converged.

Ctors.

Loads a fragment from memory with additional logical offset

Used to guarantee that the Pipeline init is visible to all producers and consumer threadblocks in the cluster

Synchronization call. Blocks until barriers are initialized in shared memory.

Stores a fragment to memory with additional pointer offset

Helper to rematerialize block Idx. Reduces register liveness.

Loads a fragment from memory with additional logical offset

Loads a fragment from memory at the location pointed to by the iterator. (scalar loads)

Computes padding in shared memory to perform efficient transpose without bank conflicts.

Number of SK blocks per region (splitting factor)

Number of SK regions

Returns the stride of the layout

Returns the stride of the layout

Returns the GEMM volume in thread block tiles

Transform the mma operands to the required types

When the work tile is not special for reduction, it's valid. Otherwise need to skip global loading that producer warpgroup do, also math computation t

Wait on a stage to be unlocked

Adds a tile offset in the unit of tile.

Computes the grid shape

Overrides the internal iteration index

Adds a pointer offset in units of Element

Helper to add an offset in number of elements, assuming this offset is divisible by the vector size.

Increment and return an instance to self.

Returns whether the block assigned this work should compute the epilogue for the corresponding output tile. For the basic tile scheduler, this is alwa

Helps with visibility of barrier init operations across warps / cta / cluster Available as a separate function so as to batch inits across barriers an

This is the method we expect to use going forward

Loads a fragment from memory with additional logical offset

prepare the MMA accumulators when initialization or zeroing is required.

Replace address for the global tensor (to be done by single thread)

Adds a pointer offset in units of Element

Returns true if source is needed based on state of runtime arguments

Perform a Consumer Epilogue to release all buffers

Stores a fragment to memory at the location pointed to by the iterator

Adds a pointer offset in units of Element

Adds a tile offset

Adds a tile offset