| 2190 | |
| 2191 | |
| 2192 | class Softmax(LayerBase): |
| 2193 | def __init__(self, dim=-1, optimizer=None): |
| 2194 | r""" |
| 2195 | A softmax nonlinearity layer. |
| 2196 | |
| 2197 | Notes |
| 2198 | ----- |
| 2199 | This is implemented as a layer rather than an activation primarily |
| 2200 | because it requires retaining the layer input in order to compute the |
| 2201 | softmax gradients properly. In other words, in contrast to other |
| 2202 | simple activations, the softmax function and its gradient are not |
| 2203 | computed elementwise, and thus are more easily expressed as a layer. |
| 2204 | |
| 2205 | The softmax function computes: |
| 2206 | |
| 2207 | .. math:: |
| 2208 | |
| 2209 | y_i = \frac{e^{x_i}}{\sum_j e^{x_j}} |
| 2210 | |
| 2211 | where :math:`x_i` is the `i` th element of input example **x**. |
| 2212 | |
| 2213 | Parameters |
| 2214 | ---------- |
| 2215 | dim: int |
| 2216 | The dimension in `X` along which the softmax will be computed. |
| 2217 | Default is -1. |
| 2218 | optimizer : str, :doc:`Optimizer <numpy_ml.neural_nets.optimizers>` object, or None |
| 2219 | The optimization strategy to use when performing gradient updates |
| 2220 | within the :meth:`update` method. If None, use the :class:`SGD |
| 2221 | <numpy_ml.neural_nets.optimizers.SGD>` optimizer with |
| 2222 | default parameters. Default is None. Unused for this layer. |
| 2223 | |
| 2224 | Attributes |
| 2225 | ---------- |
| 2226 | X : list |
| 2227 | Running list of inputs to the :meth:`forward <numpy_ml.neural_nets.LayerBase.forward>` method since the last call to :meth:`update <numpy_ml.neural_nets.LayerBase.update>`. Only updated if the `retain_derived` argument was set to True. |
| 2228 | gradients : dict |
| 2229 | Dictionary of loss gradients with regard to the layer parameters |
| 2230 | parameters : dict |
| 2231 | Dictionary of layer parameters |
| 2232 | hyperparameters : dict |
| 2233 | Dictionary of layer hyperparameters |
| 2234 | derived_variables : dict |
| 2235 | Dictionary of any intermediate values computed during |
| 2236 | forward/backward propagation. |
| 2237 | """ # noqa: E501 |
| 2238 | super().__init__(optimizer) |
| 2239 | |
| 2240 | self.dim = dim |
| 2241 | self.n_in = None |
| 2242 | self.is_initialized = False |
| 2243 | |
| 2244 | def _init_params(self): |
| 2245 | self.gradients = {} |
| 2246 | self.parameters = {} |
| 2247 | self.derived_variables = {} |
| 2248 | self.is_initialized = True |
| 2249 |
no outgoing calls