Forward pass through the RMSNorm layer. Args: x (torch.Tensor): The input tensor. Returns: torch.Tensor: The output tensor after applying RMSNorm.
(self, x)
| 39 | return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) |
| 40 | |
| 41 | def forward(self, x): |
| 42 | """ |
| 43 | Forward pass through the RMSNorm layer. |
| 44 | |
| 45 | Args: |
| 46 | x (torch.Tensor): The input tensor. |
| 47 | |
| 48 | Returns: |
| 49 | torch.Tensor: The output tensor after applying RMSNorm. |
| 50 | |
| 51 | """ |
| 52 | output = self._norm(x.float()).type_as(x) |
| 53 | return output * self.weight |
| 54 | |
| 55 | |
| 56 |