Implements stochastic gradient descent (optionally with momentum). Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__. Args: lr(float): learning rate momentum(float, optional): momentum factor(default: 0)
| 57 | |
| 58 | # MSSGD -- actually no change of code |
| 59 | class MSSGD(MSOptimizer): |
| 60 | """Implements stochastic gradient descent (optionally with momentum). |
| 61 | Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__. |
| 62 | Args: |
| 63 | lr(float): learning rate |
| 64 | momentum(float, optional): momentum factor(default: 0) |
| 65 | weight_decay(float, optional): weight decay(L2 penalty)(default: 0) |
| 66 | dampening(float, optional): dampening for momentum(default: 0) |
| 67 | nesterov(bool, optional): enables Nesterov momentum(default: False) |
| 68 | Typical usage example: |
| 69 | >> > from singa import opt |
| 70 | >> > optimizer = opt.SGD(lr=0.1, momentum=0.9) |
| 71 | >> > optimizer.update() |
| 72 | __ http: // www.cs.toronto.edu / %7Ehinton / absps / momentum.pdf |
| 73 | .. note:: |
| 74 | The implementation of SGD with Momentum / Nesterov subtly differs from |
| 75 | Sutskever et. al. and implementations in some other frameworks. |
| 76 | Considering the specific case of Momentum, the update can be written as |
| 77 | .. math:: |
| 78 | v = \rho * v + g \\ |
| 79 | p = p - lr * v |
| 80 | where p, g, v and: math: `\rho` denote the parameters, gradient, |
| 81 | velocity, and momentum respectively. |
| 82 | This is in contrast to Sutskever et. al. and |
| 83 | other frameworks which employ an update of the form |
| 84 | .. math:: |
| 85 | v = \rho * v + lr * g \\ |
| 86 | p = p - v |
| 87 | The Nesterov version is analogously modified. |
| 88 | """ |
| 89 | |
| 90 | def __init__(self, |
| 91 | lr=0.1, |
| 92 | momentum=0, |
| 93 | dampening=0, |
| 94 | weight_decay=0, |
| 95 | nesterov=False, |
| 96 | dtype=tensor.float32): |
| 97 | super(MSSGD, self).__init__(lr, dtype) |
| 98 | |
| 99 | # init momentum |
| 100 | if type(momentum) == float or type(momentum) == int: |
| 101 | if momentum < 0.0: |
| 102 | raise ValueError("Invalid momentum value: {}".format(momentum)) |
| 103 | self.momentum = Constant(momentum) |
| 104 | elif isinstance(momentum, DecayScheduler): |
| 105 | self.momentum = momentum |
| 106 | momentum = momentum.init_value |
| 107 | else: |
| 108 | raise TypeError("Wrong momentum type") |
| 109 | self.mom_value = self.momentum(self.step_counter).as_type(self.dtype) |
| 110 | |
| 111 | # init dampening |
| 112 | if type(dampening) == float or type(dampening) == int: |
| 113 | self.dampening = Constant(dampening) |
| 114 | elif isinstance(dampening, DecayScheduler): |
| 115 | self.dampening = dampening |
| 116 | dampening = dampening.init_value |