Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT). Also see https://arxiv.org/abs/1606.08415
(x)
| 23 | |
| 24 | |
| 25 | def gelu_new(x): |
| 26 | """ Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT). |
| 27 | Also see https://arxiv.org/abs/1606.08415 |
| 28 | """ |
| 29 | return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0)))) |
| 30 | |
| 31 | |
| 32 | if torch.__version__ < "1.4.0": |
nothing calls this directly
no outgoing calls
no test coverage detected