Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__. Args: lr(float): learning rate momentum(float, optional): momentum factor(default: 0)
| 52 | |
| 53 | # MSSGD -- actually no change of code |
| 54 | class MSSGD(MSOptimizer): |
| 55 | """Implements stochastic gradient descent (optionally with momentum). |
| 56 | |
| 57 | Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__. |
| 58 | |
| 59 | Args: |
| 60 | lr(float): learning rate |
| 61 | momentum(float, optional): momentum factor(default: 0) |
| 62 | weight_decay(float, optional): weight decay(L2 penalty)(default: 0) |
| 63 | dampening(float, optional): dampening for momentum(default: 0) |
| 64 | nesterov(bool, optional): enables Nesterov momentum(default: False) |
| 65 | |
| 66 | Typical usage example: |
| 67 | >> > from singa import opt |
| 68 | >> > optimizer = opt.SGD(lr=0.1, momentum=0.9) |
| 69 | >> > optimizer.update() |
| 70 | |
| 71 | __ http: // www.cs.toronto.edu / %7Ehinton / absps / momentum.pdf |
| 72 | |
| 73 | .. note:: |
| 74 | The implementation of SGD with Momentum / Nesterov subtly differs from |
| 75 | Sutskever et. al. and implementations in some other frameworks. |
| 76 | |
| 77 | Considering the specific case of Momentum, the update can be written as |
| 78 | |
| 79 | .. math:: |
| 80 | v = \rho * v + g \\ |
| 81 | p = p - lr * v |
| 82 | |
| 83 | where p, g, v and: math: `\rho` denote the parameters, gradient, |
| 84 | velocity, and momentum respectively. |
| 85 | |
| 86 | This is in contrast to Sutskever et. al. and |
| 87 | other frameworks which employ an update of the form |
| 88 | |
| 89 | .. math:: |
| 90 | v = \rho * v + lr * g \\ |
| 91 | p = p - v |
| 92 | |
| 93 | The Nesterov version is analogously modified. |
| 94 | """ |
| 95 | |
| 96 | def __init__(self, |
| 97 | lr=0.1, |
| 98 | momentum=0, |
| 99 | dampening=0, |
| 100 | weight_decay=0, |
| 101 | nesterov=False, |
| 102 | dtype=tensor.float32): |
| 103 | super(MSSGD, self).__init__(lr, dtype) |
| 104 | |
| 105 | # init momentum |
| 106 | if type(momentum) == float or type(momentum) == int: |
| 107 | if momentum < 0.0: |
| 108 | raise ValueError("Invalid momentum value: {}".format(momentum)) |
| 109 | self.momentum = Constant(momentum) |
| 110 | elif isinstance(momentum, DecayScheduler): |
| 111 | self.momentum = momentum |