Computes sparsemax activations [1]. For each batch `i` and class `j` we have $$sparsemax[i, j] = max(logits[i, j] - tau(logits[i, :]), 0)$$ [1]: https://arxiv.org/abs/1602.02068 Args: logits: A `Tensor`. Must be one of the following types: `half`, `float32`, `float64`. nam
(logits, name=None)
| 28 | |
| 29 | |
| 30 | def sparsemax(logits, name=None): |
| 31 | """Computes sparsemax activations [1]. |
| 32 | |
| 33 | For each batch `i` and class `j` we have |
| 34 | $$sparsemax[i, j] = max(logits[i, j] - tau(logits[i, :]), 0)$$ |
| 35 | |
| 36 | [1]: https://arxiv.org/abs/1602.02068 |
| 37 | |
| 38 | Args: |
| 39 | logits: A `Tensor`. Must be one of the following types: `half`, `float32`, |
| 40 | `float64`. |
| 41 | name: A name for the operation (optional). |
| 42 | |
| 43 | Returns: |
| 44 | A `Tensor`. Has the same type as `logits`. |
| 45 | """ |
| 46 | |
| 47 | with ops.name_scope(name, "sparsemax", [logits]) as name: |
| 48 | logits = ops.convert_to_tensor(logits, name="logits") |
| 49 | obs = array_ops.shape(logits)[0] |
| 50 | dims = array_ops.shape(logits)[1] |
| 51 | |
| 52 | # In the paper, they call the logits z. |
| 53 | # The mean(logits) can be substracted from logits to make the algorithm |
| 54 | # more numerically stable. the instability in this algorithm comes mostly |
| 55 | # from the z_cumsum. Substacting the mean will cause z_cumsum to be close |
| 56 | # to zero. However, in practise the numerical instability issues are very |
| 57 | # minor and substacting the mean causes extra issues with inf and nan |
| 58 | # input. |
| 59 | z = logits |
| 60 | |
| 61 | # sort z |
| 62 | z_sorted, _ = nn.top_k(z, k=dims) |
| 63 | |
| 64 | # calculate k(z) |
| 65 | z_cumsum = math_ops.cumsum(z_sorted, axis=1) |
| 66 | k = math_ops.range( |
| 67 | 1, math_ops.cast(dims, logits.dtype) + 1, dtype=logits.dtype) |
| 68 | z_check = 1 + k * z_sorted > z_cumsum |
| 69 | # because the z_check vector is always [1,1,...1,0,0,...0] finding the |
| 70 | # (index + 1) of the last `1` is the same as just summing the number of 1. |
| 71 | k_z = math_ops.reduce_sum(math_ops.cast(z_check, dtypes.int32), axis=1) |
| 72 | |
| 73 | # calculate tau(z) |
| 74 | # If there are inf values or all values are -inf, the k_z will be zero, |
| 75 | # this is mathematically invalid and will also cause the gather_nd to fail. |
| 76 | # Prevent this issue for now by setting k_z = 1 if k_z = 0, this is then |
| 77 | # fixed later (see p_safe) by returning p = nan. This results in the same |
| 78 | # behavior as softmax. |
| 79 | k_z_safe = math_ops.maximum(k_z, 1) |
| 80 | indices = array_ops.stack([math_ops.range(0, obs), k_z_safe - 1], axis=1) |
| 81 | tau_sum = array_ops.gather_nd(z_cumsum, indices) |
| 82 | tau_z = (tau_sum - 1) / math_ops.cast(k_z, logits.dtype) |
| 83 | |
| 84 | # calculate p |
| 85 | p = math_ops.maximum( |
| 86 | math_ops.cast(0, logits.dtype), z - tau_z[:, array_ops.newaxis]) |
| 87 | # If k_z = 0 or if z = nan, then the input is invalid |