Number of bytes in the input, not counting the attention bias.

Number of bytes in the input, not counting the attention bias.

Call into Triton's upstream cost model, with the right args The upstream function expects arguments to have certain names. Since we renamed a

r""" Given a mask pattern and blocksize, return the corresponding layout which makes sure that all the positives in the mask are covered

Graph out the contents of a dict. Dash key means that if the result label has this key, then it will be displayed with a dash

A pre-configured profiler that will run on the first ~20 steps of the training It will provide multiple traces that can be exploited later.

* generate a tensor with random uniform values. only used for testing, not much * attention is paid to performance */

Registers a subclass. This decorator allows xFormers to instantiate a given subclass from a configuration file, even if the class its

In-place scaling+index_add Indices in ``index`` are assumed to be unique The max index in ``index`` is assumed to be less than the size

An activation checkpoint context_fn for selectively deciding what to store and what to recompute. Accepts a custom policy. Args: polic

2:4 sparsification, with Straight Through Estimator for the backward pass (eg the gradient is *not* sparsified). Optionally, `bw_mul[0-1]

The inverse operation of :attr:`BlockDiagonalCausalMask.from_tensor_list` Args: tensor (torch.Tensor): Tensor of tokens of shape

See `xformers.profiler.profile`