Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... Se
(x, drop_prob: float = 0., training: bool = False, scale_by_keep: bool = True)
| 2 | import torch |
| 3 | |
| 4 | def drop_path(x, drop_prob: float = 0., training: bool = False, scale_by_keep: bool = True): |
| 5 | """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). |
| 6 | |
| 7 | This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, |
| 8 | the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... |
| 9 | See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for |
| 10 | changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use |
| 11 | 'survival rate' as the argument. |
| 12 | |
| 13 | """ |
| 14 | if drop_prob == 0. or not training: |
| 15 | return x |
| 16 | keep_prob = 1 - drop_prob |
| 17 | shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets |
| 18 | random_tensor = x.new_empty(shape).bernoulli_(keep_prob) |
| 19 | if keep_prob > 0.0 and scale_by_keep: |
| 20 | random_tensor.div_(keep_prob) |
| 21 | return x * random_tensor |
| 22 | |
| 23 | |
| 24 | class DropPath(torch.nn.Module): |