If no closure is supplied, :attr:`step` should be called after ``fp16_optimizer_obj.backward(loss)``. :attr:`step` updates the fp32 master copy of parameters using the optimizer supplied to :class:`FP16_Optimizer`'s constructor, then copies the updated fp32 params i
(self, closure=None)
| 397 | current.data.copy_(saved.data) |
| 398 | |
| 399 | def step(self, closure=None): # could add clip option. |
| 400 | """ |
| 401 | If no closure is supplied, :attr:`step` should be called after |
| 402 | ``fp16_optimizer_obj.backward(loss)``. |
| 403 | :attr:`step` updates the fp32 master copy of parameters using the optimizer supplied to |
| 404 | :class:`FP16_Optimizer`'s constructor, then copies the updated fp32 params into the fp16 params |
| 405 | originally referenced by :class:`FP16_Optimizer`'s constructor, so the user may immediately run |
| 406 | another forward pass using their model. |
| 407 | |
| 408 | If a closure is supplied, :attr:`step` may be called without a prior call to |
| 409 | :attr:`backward(loss)`. |
| 410 | This control flow is identical to `ordinary Pytorch optimizer use`_ with closures. |
| 411 | However, the user should take care that any ``loss.backward()`` call within the closure |
| 412 | has been replaced by ``fp16_optimizer_obj.backward(loss)``. |
| 413 | |
| 414 | Args: |
| 415 | closure (optional): Closure that will be supplied to the underlying optimizer originally passed to :class:`FP16_Optimizer`'s constructor. closure should call :attr:`zero_grad()` on the :class:`FP16_Optimizer` object, compute the loss, call :attr:`backward(loss)`, and return the loss. |
| 416 | |
| 417 | Example with closure:: |
| 418 | |
| 419 | # optimizer is assumed to be an FP16_Optimizer object, previously constructed from an |
| 420 | # existing pytorch optimizer. |
| 421 | for input, target in dataset: |
| 422 | def closure(): |
| 423 | optimizer.zero_grad() |
| 424 | output = model(input) |
| 425 | loss = loss_fn(output, target) |
| 426 | # loss.backward() becomes: |
| 427 | optimizer.backward(loss) |
| 428 | return loss |
| 429 | optimizer.step(closure) |
| 430 | |
| 431 | .. warning:: |
| 432 | Currently, calling :attr:`step` with a closure is not compatible with dynamic loss scaling. |
| 433 | |
| 434 | .. _`ordinary Pytorch optimizer use`: |
| 435 | http://pytorch.org/docs/master/optim.html#optimizer-step-closure |
| 436 | """ |
| 437 | |
| 438 | scale = self.loss_scaler.loss_scale |
| 439 | self._update_scale(self.overflow) |
| 440 | |
| 441 | if self.overflow: |
| 442 | self.maybe_print("OVERFLOW! Skipping step. Attempted loss scale: {}, reducing to {}" |
| 443 | .format(scale, self.loss_scale)) |
| 444 | return |
| 445 | |
| 446 | if closure is not None: |
| 447 | retval = self._step_with_closure(closure) |
| 448 | else: |
| 449 | retval = self.optimizer.step() |
| 450 | |
| 451 | self._master_params_to_model_params() |
| 452 | |
| 453 | return retval |
| 454 | |
| 455 | def _step_with_closure(self, closure): |
| 456 | def wrapped_closure(): |
no test coverage detected