Apply a regularizer on trainable variables matching the regex, and print the matched variables (only print once in multi-tower training). In replicated mode, it will only regularize variables within the current tower. If called under a TowerContext with `is_training==False`, this f
(regex, func, name='regularize_cost')
| 31 | |
| 32 | |
| 33 | def regularize_cost(regex, func, name='regularize_cost'): |
| 34 | """ |
| 35 | Apply a regularizer on trainable variables matching the regex, and print |
| 36 | the matched variables (only print once in multi-tower training). |
| 37 | In replicated mode, it will only regularize variables within the current tower. |
| 38 | |
| 39 | If called under a TowerContext with `is_training==False`, this function returns a zero constant tensor. |
| 40 | |
| 41 | Args: |
| 42 | regex (str): a regex to match variable names, e.g. "conv.*/W" |
| 43 | func: the regularization function, which takes a tensor and returns a scalar tensor. |
| 44 | E.g., ``tf.nn.l2_loss, tf.contrib.layers.l1_regularizer(0.001)``. |
| 45 | |
| 46 | Returns: |
| 47 | tf.Tensor: a scalar, the total regularization cost. |
| 48 | |
| 49 | Example: |
| 50 | .. code-block:: python |
| 51 | |
| 52 | cost = cost + regularize_cost("fc.*/W", l2_regularizer(1e-5)) |
| 53 | """ |
| 54 | assert len(regex) |
| 55 | ctx = get_current_tower_context() |
| 56 | if not ctx.is_training: |
| 57 | # Currently cannot build the wd_cost correctly at inference, |
| 58 | # because ths vs_name used in inference can be '', therefore the |
| 59 | # variable filter will fail |
| 60 | return tf.constant(0, dtype=tf.float32, name='empty_' + name) |
| 61 | |
| 62 | # If vars are shared, regularize all of them |
| 63 | # If vars are replicated, only regularize those in the current tower |
| 64 | if ctx.has_own_variables: |
| 65 | params = ctx.get_collection_in_tower(tfv1.GraphKeys.TRAINABLE_VARIABLES) |
| 66 | else: |
| 67 | params = tfv1.trainable_variables() |
| 68 | |
| 69 | names = [] |
| 70 | |
| 71 | with tfv1.name_scope(name + '_internals'): |
| 72 | costs = [] |
| 73 | for p in params: |
| 74 | para_name = p.op.name |
| 75 | if re.search(regex, para_name): |
| 76 | regloss = func(p) |
| 77 | assert regloss.dtype.is_floating, regloss |
| 78 | # Some variables may not be fp32, but it should |
| 79 | # be fine to assume regularization in fp32 |
| 80 | if regloss.dtype != tf.float32: |
| 81 | regloss = tf.cast(regloss, tf.float32) |
| 82 | costs.append(regloss) |
| 83 | names.append(p.name) |
| 84 | if not costs: |
| 85 | return tf.constant(0, dtype=tf.float32, name='empty_' + name) |
| 86 | |
| 87 | # remove tower prefix from names, and print |
| 88 | if len(ctx.vs_name): |
| 89 | prefix = ctx.vs_name + '/' |
| 90 | prefixlen = len(prefix) |
no test coverage detected