Forward pass of the BitLinear layer. Args: x (Tensor): The input tensor. Returns: Tensor: The output tensor.
(self, x: Tensor)
| 40 | """ |
| 41 | |
| 42 | def forward(self, x: Tensor) -> Tensor: |
| 43 | """ |
| 44 | Forward pass of the BitLinear layer. |
| 45 | |
| 46 | Args: |
| 47 | x (Tensor): The input tensor. |
| 48 | |
| 49 | Returns: |
| 50 | Tensor: The output tensor. |
| 51 | |
| 52 | """ |
| 53 | w = self.weight |
| 54 | x_norm = SimpleRMSNorm(self.in_features)(x) |
| 55 | |
| 56 | # STE using detach |
| 57 | x_quant = x_norm + (activation_quant(x_norm) - x_norm).detach() |
| 58 | w_quant = w + (weight_quant(w) - w).detach() |
| 59 | y = F.linear(x_quant, w_quant) |
| 60 | return y |
nothing calls this directly
no test coverage detected