Computes alpha dropout. Alpha Dropout is a dropout that maintains the self-normalizing property. For an input with zero mean and unit standard deviation, the output of Alpha Dropout maintains the original mean and standard deviation of the input. See [Self-Normalizing Neural Networks](http
(x, keep_prob, noise_shape=None, seed=None, name=None)
| 27 | |
| 28 | |
| 29 | def alpha_dropout(x, keep_prob, noise_shape=None, seed=None, name=None): # pylint: disable=invalid-name |
| 30 | """Computes alpha dropout. |
| 31 | |
| 32 | Alpha Dropout is a dropout that maintains the self-normalizing property. For |
| 33 | an input with zero mean and unit standard deviation, the output of |
| 34 | Alpha Dropout maintains the original mean and standard deviation of the input. |
| 35 | |
| 36 | See [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515) |
| 37 | |
| 38 | Args: |
| 39 | x: A tensor. |
| 40 | keep_prob: A scalar `Tensor` with the same type as x. The probability |
| 41 | that each element is kept. |
| 42 | noise_shape: A 1-D `Tensor` of type `int32`, representing the |
| 43 | shape for randomly generated keep/drop flags. |
| 44 | seed: A Python integer. Used to create random seeds. See |
| 45 | `tf.compat.v1.set_random_seed` for behavior. |
| 46 | name: A name for this operation (optional). |
| 47 | |
| 48 | Returns: |
| 49 | A Tensor of the same shape of `x`. |
| 50 | |
| 51 | Raises: |
| 52 | ValueError: If `keep_prob` is not in `(0, 1]`. |
| 53 | |
| 54 | """ |
| 55 | with ops.name_scope(name, "alpha_dropout", [x]) as name: |
| 56 | x = ops.convert_to_tensor(x, name="x") |
| 57 | if isinstance(keep_prob, numbers.Real) and not 0 < keep_prob <= 1.: |
| 58 | raise ValueError("keep_prob must be a scalar tensor or a float in the " |
| 59 | "range (0, 1], got %g" % keep_prob) |
| 60 | keep_prob = ops.convert_to_tensor(keep_prob, |
| 61 | dtype=x.dtype, |
| 62 | name="keep_prob") |
| 63 | keep_prob.get_shape().assert_has_rank(0) |
| 64 | |
| 65 | # Do nothing if we know keep_prob == 1 |
| 66 | if tensor_util.constant_value(keep_prob) == 1: |
| 67 | return x |
| 68 | |
| 69 | alpha = -1.7580993408473766 |
| 70 | |
| 71 | noise_shape = noise_shape if noise_shape is not None else array_ops.shape(x) |
| 72 | random_tensor = random_ops.random_uniform(noise_shape, |
| 73 | seed=seed, |
| 74 | dtype=x.dtype) |
| 75 | kept_idx = gen_math_ops.greater_equal(random_tensor, 1 - keep_prob) |
| 76 | kept_idx = math_ops.cast(kept_idx, x.dtype) |
| 77 | # Mask |
| 78 | x = x * kept_idx + alpha * (1 - kept_idx) |
| 79 | |
| 80 | # Affine transformation parameters |
| 81 | a = (keep_prob + keep_prob * (1 - keep_prob) * alpha ** 2) ** -0.5 |
| 82 | b = -a * alpha * (1 - keep_prob) |
| 83 | |
| 84 | # Affine transformation |
| 85 | return a * x + b |