Given a superset of the inputs and a reference config class, return exactly the needed config

Returns a triton.runtime.autotuner.AutoTuner.cache object, which represents mappings from kernel autotune keys (tuples describing kernel input

Number of nodes in a full tree of a given depth (including the root node) and branching factor.

Indices in ``index`` are assumed to be unique In each (index, source) pair, the max index in ``index`` is assumed to be less than the size of

Initialize model parallelism support. Args: world_size (int): the number of processes running on the current node availa

For testing xFormers building and binding

r""" create a pattern of shape [heads, seq, seq] out of a blocksparse layout of shape [heads, seq/block_size, seq/block_size]

Allocate a cache to be used with the Transformer module. Args: args (ModelArgs): the model configuration. length (int): per

Create a new bias which combines local + causal attention. See :attr:`LowerTriangularFromBottomRightLocalAttentionMask`

Materialize the attention bias - for debugging & testing

Materialize the attention bias - for debugging & testing

Materialize the attention bias - for debugging & testing

Materialize the attention bias - for debugging & testing

Materialize the attention bias - for debugging & testing

Materialize the attention bias - for debugging & testing

Implements the memory-efficient attention mechanism following `"Self-Attention Does Not Need O(n^2) Memory" <http://arxiv.org/abs/2112.05682>`_.

Computes the gradient of the attention. Returns a tuple (dq, dk, dv) See :attr:`xformers.ops.memory_efficient_attention` for an explanati

This provides a torch-compilable wrapper op to memory_efficient_attention_forward in certain special cases. Note that the following are