Adds a Batch Normalization layer from http://arxiv.org/abs/1502.03167. "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" Sergey Ioffe, Christian Szegedy Can be used as a normalizer function for conv2d and fully_connected. The normalizat
(inputs,
decay=0.999,
center=True,
scale=False,
epsilon=0.001,
activation_fn=None,
param_initializers=None,
param_regularizers=None,
updates_collections=ops.GraphKeys.UPDATE_OPS,
is_training=True,
reuse=None,
variables_collections=None,
outputs_collections=None,
trainable=True,
batch_weights=None,
fused=None,
data_format=DATA_FORMAT_NHWC,
zero_debias_moving_mean=False,
scope=None,
renorm=False,
renorm_clipping=None,
renorm_decay=0.99,
adjustment=None)
| 432 | |
| 433 | @add_arg_scope |
| 434 | def batch_norm(inputs, |
| 435 | decay=0.999, |
| 436 | center=True, |
| 437 | scale=False, |
| 438 | epsilon=0.001, |
| 439 | activation_fn=None, |
| 440 | param_initializers=None, |
| 441 | param_regularizers=None, |
| 442 | updates_collections=ops.GraphKeys.UPDATE_OPS, |
| 443 | is_training=True, |
| 444 | reuse=None, |
| 445 | variables_collections=None, |
| 446 | outputs_collections=None, |
| 447 | trainable=True, |
| 448 | batch_weights=None, |
| 449 | fused=None, |
| 450 | data_format=DATA_FORMAT_NHWC, |
| 451 | zero_debias_moving_mean=False, |
| 452 | scope=None, |
| 453 | renorm=False, |
| 454 | renorm_clipping=None, |
| 455 | renorm_decay=0.99, |
| 456 | adjustment=None): |
| 457 | """Adds a Batch Normalization layer from http://arxiv.org/abs/1502.03167. |
| 458 | |
| 459 | "Batch Normalization: Accelerating Deep Network Training by Reducing |
| 460 | Internal Covariate Shift" |
| 461 | |
| 462 | Sergey Ioffe, Christian Szegedy |
| 463 | |
| 464 | Can be used as a normalizer function for conv2d and fully_connected. The |
| 465 | normalization is over all but the last dimension if `data_format` is `NHWC` |
| 466 | and all but the second dimension if `data_format` is `NCHW`. In case of a 2D |
| 467 | tensor this corresponds to the batch dimension, while in case of a 4D tensor |
| 468 | this |
| 469 | corresponds to the batch and space dimensions. |
| 470 | |
| 471 | Note: when training, the moving_mean and moving_variance need to be updated. |
| 472 | By default the update ops are placed in `tf.GraphKeys.UPDATE_OPS`, so they |
| 473 | need to be added as a dependency to the `train_op`. For example: |
| 474 | |
| 475 | ```python |
| 476 | update_ops = tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS) |
| 477 | with tf.control_dependencies(update_ops): |
| 478 | train_op = optimizer.minimize(loss) |
| 479 | ``` |
| 480 | |
| 481 | One can set updates_collections=None to force the updates in place, but that |
| 482 | can have a speed penalty, especially in distributed settings. |
| 483 | |
| 484 | Args: |
| 485 | inputs: A tensor with 2 or more dimensions, where the first dimension has |
| 486 | `batch_size`. The normalization is over all but the last dimension if |
| 487 | `data_format` is `NHWC` and the second dimension if `data_format` is |
| 488 | `NCHW`. |
| 489 | decay: Decay for the moving average. Reasonable values for `decay` are close |
| 490 | to 1.0, typically in the multiple-nines range: 0.999, 0.99, 0.9, etc. |
| 491 | Lower `decay` value (recommend trying `decay`=0.9) if model experiences |