This is used to overwrite the gradients with a new gradient vector, whenever violations occur. pp: parameters newgrad: corrected gradient grad_dims: list storing number of parameters at each layer
(pp, newgrad, grad_dims)
| 43 | |
| 44 | |
| 45 | def overwrite_grad(pp, newgrad, grad_dims): |
| 46 | """ |
| 47 | This is used to overwrite the gradients with a new gradient |
| 48 | vector, whenever violations occur. |
| 49 | pp: parameters |
| 50 | newgrad: corrected gradient |
| 51 | grad_dims: list storing number of parameters at each layer |
| 52 | """ |
| 53 | cnt = 0 |
| 54 | for param in pp(): |
| 55 | if param.grad is not None: |
| 56 | beg = 0 if cnt == 0 else sum(grad_dims[:cnt]) |
| 57 | en = sum(grad_dims[:cnt + 1]) |
| 58 | this_grad = newgrad[beg: en].contiguous().view( |
| 59 | param.grad.data.size()) |
| 60 | param.grad.data.copy_(this_grad) |
| 61 | cnt += 1 |
| 62 | |
| 63 | |
| 64 | def project2cone2(gradient, memories, margin=0.5, eps=1e-3): |