Method forward

bitnet/one_bit_vision_transformers.py:40–42

(self, x)

        self.gamma = nn.Parameter(torch.ones(heads, 1, dim) / self.scale)

    def forward(self, x):
        normed = F.normalize(x, dim=-1)
        return normed * self.scale * self.gamma


# classes