Input tensors are assumed to be in shape [B, M, *]

Materializes the bias as a `torch.Tensor`. This is very slow and we don't attempt to make it fast. Only use for debugging/testing.

Wrapper around torch.utils.checkpoint that accepts a custom policy function for selectively deciding what to store and what to recompute Args:

Input tensors are assumed to be in shape [B, M, *] seqstart = padding * torch.arange(batch_size)

Specializes a triton kernel with variable number of inputs to a specific number of inputs `N`. `mode` can either be `UNROLL` or `CONDITI

call this before entering a new FW pass

Args: tree_choices: tree description in the style of https://github.com/FasterDecoding/Medusa/blob/5e9805386/medu

Assuming our keys actually live in separate blocks of length notional_padding, convert to a Paged version, avoiding GPU syncs.

Wrapper around `memory_efficient_attention_partial` which works with autograd. Arguments are the same as for `memory_efficient_attention_part

Wrapper around merge_attentions which works with autograd.

Signals the profiler that the next profiling step has started.

Performs RoPE (rotary embeddings) and kv-cache emplacement for a heterogeneous batch for inference in the style given by BlockDiagonalCau

This method determines whether to use normal or smallq splitkv kernel

Helper function to run benchmarks. Supports loading previous results for comparison, and saving current results to file.

Creates a :attr:`BlockDiagonalMask` from a list of tensors, and returns the tensors concatenated on the sequence length dimension ..

Makes each block causal with a possible non-causal prefix

Multiply two matrices given as grids of tiles It performs the matmul between A and B, which are given as two-dimensional grids of tiles (i.e.