Embeds raw inputs into the geometric algebra (+ scalar) representation. To be implemented by subclasses. Parameters --------

Embeds raw inputs into the geometric algebra (+ scalar) representation. To be implemented by subclasses. Parameters --------

Embeds raw inputs into the geometric algebra (+ scalar) representation. To be implemented by subclasses. Parameters --------

Extracts the predicted wall shear stress from the output multivectors. We parameterize the wall shear stress as translation-invariant vectors

Extracts raw outputs from the GATr multivector + scalar outputs. We parameterize the predicted final positions as points. Parameters

Forward pass of the Xformers model.

Forward pass. Parameters ---------- inputs : torch.Tensor with shape (..., 16) Input multivectors. Retur

Forward pass.

Forward pass.

Forward pass of baseline MLP.

Forward pass. Parameters ---------- inputs : Tensor Input data Returns ------- outputs :

Forward pass. Returns ------- q : Tensor Queries k : Tensor Keys v : Tensor

Forward pass. Parameters ---------- inputs : Tensor Input data Returns ------- q : Tenso

Forward pass. Parameters ---------- inputs : Tensor Input data attention_mask : None or Tensor or xformer

Forward pass. Parameters ---------- inputs : Tensor Input data attention_mask : None or Tensor or xformer

Forward pass. Parameters ---------- inputs : Tensor with shape (..., num_items, num_channels) Input data

Forward pass. Parameters ---------- inputs : Tensor with shape (..., num_items1, num_items2, num_channels) Input

Forward pass.

Compute compiled linear map. Parameters ---------- multivectors : torch.Tensor with shape (..., in_mv_channels, 16)

Forward pass of the network. Parameters ---------- multivectors : torch.Tensor with shape (..., in_mv_channels, 16)

Forward pass of the network. Parameters ---------- multivectors : Tensor with shape (..., num_items_1, num_items_2, in_mv_cha

Wrapped forward pass pass. Parses inputs into GA + scalar representation, calls the forward pass of the wrapped net, and extracts the

Wrapped forward pass.

Wrapped forward pass. Parameters ---------- inputs : torch.Tensor Raw inputs, as given by dataset. Retur

Wrapped forward pass. Parameters ---------- inputs : torch.Tensor Raw inputs, as given by dataset. Retur

Wrapped forward pass. Parameters ---------- inputs : torch.Tensor Raw inputs, as given by dataset. Retur

Wrapped forward pass. Parameters ---------- inputs : torch.Tensor Raw inputs, as given by dataset. Retur

Maps input multivectors and scalars using the most general equivariant linear map. The result is again multivectors and scalars. For

Forward pass. Applies dropout. Parameters ---------- multivectors : torch.Tensor with shape (..., 16) Multivector

Forward pass of the transformer block. Inputs are first processed by a block consisting of LayerNorm, multi-head geometric self-atten

Forward pass. Computes equivariant LayerNorm for multivectors. Parameters ---------- multivectors : torch.Tensor with shape (

Computes rotary embeddings along `self.item_dim` and applies them to inputs. The inputs are usually scalar queries and keys. Assumes

Forward pass. Parameters ---------- inputs : torch.Tensor Multivector inputs scalars : torch.Tensor

Forward pass. Parameters ---------- inputs : torch.Tensor Multivector inputs scalars : torch.Tensor

Computes forward pass on inputs with shape `(..., items, channels, 16)`. The result is the following: ``` # For each head

Forward pass through geometric attention. Given multivector and scalar queries, keys, and values, this forward pass computes: ```

Computes f(x_0) * x for multivector x, where f is GELU, ReLU, or sigmoid. f is chosen depending on self.nonlinearity. Parameters

Forward pass. Parameters ---------- multivectors : torch.Tensor with shape (..., in_mv_channels, 16) Input multiv

Forward pass. Parameters ---------- multivectors : torch.Tensor with shape (..., in_mv_channels, 16) Input multiv

Pin-equivariant gated ReLU nonlinearity. Given multivector input x and scalar input gates (with matching batch dimensions), computes ReLU(gat

Pin-equivariant gated sigmoid nonlinearity. Given multivector input x and scalar input gates (with matching batch dimensions), computes sigmo

Factory function for a GATr model.

Returns the number of hidden multivector channels.

Returns the number of hidden scalar channels.

Wrap super method with adding shape id.

Constructs message.

Decorator that ensures input tensors are autocast to a minimum precision. Only has an effect in autocast-enabled regions. Otherwise, does not cha

Multivector input fixture for testing

Computes five invariants from multivectors: scalar component, norms of the four other grades. NOTE: this primitive is not used widely in our arch

Registers a hook. Hooks are functions called either at a specific step or every n steps during training.

Rotate data sample with pre-sampled rotation matrices.

Scalar input fixture for testing

Tests whether 3D point embeddings into multivectors are cycle consistent.

Tests whether 3D point embeddings in the two-vec parameterizations into multivectors are cycle consistent.

Tests ApplyRotaryPositionalEncoding for consistency.

Tests the SelfAttention layer for Pin equivariance, with scalar inputs.

Tests the SelfAttention layer for Pin equivariance, with scalar inputs.

Tests AxialGATr for equivariance.

Tests the output shape of AxialGATr.

Tests the output shape of BaselineAxialTransformer.

Test block stacking.

Test in lienar compilation in integration. Verify that all EquiLinear submodules are substitued by CompiledLinear. Verify that the forward pa

Test equivalence of the EquiLinear and its compiled version.

Test we can properly run a backwards pass.

Test cross attention shapes.

Tests GradeDropout for equivariance.

Tests the grade_dropout() primitive for equivariance (at test time).

Tests GradeDropout for correct expectation.

Tests that grade_dropout() has train-time and test-time behaviour that is identical in expectation.

Tests GradeDropout for shape correctness.

Tests grade_dropout() map for shape correctness.

Tests that GradeDropout does nothing when p = 0.

Tests that grade_dropout() does nothing when p = 0.

Tests that dual() computes the correct dual.

Checks that torch.einsum and cached_einsum deliver the same results.

Tests that efficient_reference_dual() and explicit_reference_dual() agree.

Test whether embed and extract is the identity.

Test whether ray matches the join construction.

Tests that embed_scalar() embeds scalars into multivectors correctly.

Tests whether the output of equi_layer_norm has the correct variance.

Tests equi_layer_norm() primitive for equivariance.

Tests whether the output of EquiLayerNorm has the correct variance.

Tests EquiLayerNorm() for equivariance.

Test expand pairwise.

Tests an identity map for equivariance (testing the test).

Tests that AMP works correctly in GATr blocks

Tests that specifying block and MLP / attention checkpointing raises a ValueError.

Tests that specifying both checkpoint and checkpoint_blocks raises a ValueError.

Tests that the checkpoint_blocks kwarg raises a DeprecationWarning.

Tests GATr for equivariance.

Tests the output shape of EquiTransformer.

Tests that GATr output is invariant under seeding and saving and loading its state dict.

Test that gated_gelu_divide() is equal to gelu(x)/x.

Tests GeoMLP() for Pin equivariance.

Tests the output shape of GeoMLP().

Tests geometric_attention() for Pin equivariance.

Tests that with the right weights, geometric_attention attends to nearby points.

Tests that outputs of geometric_attention() have correct shape.

Tests GeometricBilinear() for equivariance.

Test cross attention shapes.