Add a GELU operation. Parameters: input : Tensor The input tensor on which the activation function is applied. Returns: The tensor produced by the activation layer.
(x: Tensor)
| 3348 | |
| 3349 | |
| 3350 | def gelu(x: Tensor) -> Tensor: |
| 3351 | ''' |
| 3352 | Add a GELU operation. |
| 3353 | |
| 3354 | Parameters: |
| 3355 | input : Tensor |
| 3356 | The input tensor on which the activation function is applied. |
| 3357 | |
| 3358 | Returns: |
| 3359 | The tensor produced by the activation layer. |
| 3360 | ''' |
| 3361 | return 0.5 * x * ( |
| 3362 | tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * pow(x, 3.0))) + 1.0) |
| 3363 | |
| 3364 | |
| 3365 | def geglu(x: Tensor) -> Tensor: |