Simple rope calculation of rope of one tensor Args: x: input, shape (B, M, H, K). seqpos: gives the position of each sequenc

vectorized implementation of scipy.stats.binom_test this makes our tests much faster reference: https://github.com/scipy/scipy/blob/v1.8.

Creates a new causal mask with an arbitrary ``torch.Tensor`` bias

Perform a fused all-gather followed by a linear layer

You can wait for the backward to finish with `done_semaphore.acquire()`

Benchmark backward pass for merge_attentions. Assumes "varargs" path, i.e. LSE and attention of chunks are provided as two lists of tensors,

Load a Llama or Code Llama checkpoint and return a new generator for this model.

Extract two float16 packed into one int32 as float32

Extract two float16 packed into one int32

Construct inputs for benchmarks and tests of Triton Split-k attention with fused row-wise FP8 dequantization. Quantization coefficients a

Construct a full tree of a given depth where each node (except for leaves) has a given number of children. The format is compatible with that

PACKED_PER_VAL is the number of values packed into each element x_. For example, for int4 quantization and x_ of type int32, PACKED_PER_VAL is 8.

Applies the custom dequantization function provided at the tensor's creation. After dequantization, the data is cast back to its orig

PACKED_PER_VAL is the number of values packed into each element x_. For example, for int4 quantization and x_ of type int32, PACKED_PER_VAL is 8.

Raises an ImportError if the current PyTorch version has an unexpected Flash-Attention.

return estimated running time in ms = max(compute, loading) + store

A more compact way to define a triton.Config, so it fits on one line

A more compact way to define a triton.Config, so it fits on one line

generate a list of numbers as num_splits to consider, the list of numbers is like 1, 2, 4, 8, 16, 32, 64, 96, 128, 160

Currently only implemented for GPUs