Perform a forward pass through the TransformerBlock. Args: x (torch.Tensor): Input tensor. freqs_cis (torch.Tensor): Precomputed cosine and sine frequencies. Returns: torch.Tensor: Output tensor after applying attention and feedforward l
(
self,
x: torch.Tensor,
freqs_cis: torch.Tensor,
)
| 319 | self.weight_init_std = 0.02 / (2 * self.num_layers) ** 0.5 |
| 320 | |
| 321 | def forward( |
| 322 | self, |
| 323 | x: torch.Tensor, |
| 324 | freqs_cis: torch.Tensor, |
| 325 | ): |
| 326 | """ |
| 327 | Perform a forward pass through the TransformerBlock. |
| 328 | |
| 329 | Args: |
| 330 | x (torch.Tensor): Input tensor. |
| 331 | freqs_cis (torch.Tensor): Precomputed cosine and sine frequencies. |
| 332 | |
| 333 | Returns: |
| 334 | torch.Tensor: Output tensor after applying attention and feedforward layers. |
| 335 | |
| 336 | """ |
| 337 | h = x + self.attention(self.attention_norm(x), freqs_cis) |
| 338 | out = h + self.feed_forward(self.ffn_norm(h)) |
| 339 | return out |
| 340 | |
| 341 | def init_weights(self): |
| 342 | for norm in (self.attention_norm, self.ffn_norm): |
nothing calls this directly
no outgoing calls
no test coverage detected