MCPcopy Create free account
hub / github.com/google-research/language / create_optimizer

Function create_optimizer

language/canine/bert_optimization.py:24–83  ·  view source on GitHub ↗

Creates an optimizer training op.

(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu)

Source from the content-addressed store, hash-verified

22
23
24def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu):
25 """Creates an optimizer training op."""
26 global_step = tf.train.get_or_create_global_step()
27
28 learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32)
29
30 # Implements linear decay of the learning rate.
31 learning_rate = tf.train.polynomial_decay(
32 learning_rate,
33 global_step,
34 num_train_steps,
35 end_learning_rate=0.0,
36 power=1.0,
37 cycle=False)
38
39 # Implements linear warmup. I.e., if global_step < num_warmup_steps, the
40 # learning rate will be `global_step/num_warmup_steps * init_lr`.
41 if num_warmup_steps:
42 global_steps_int = tf.cast(global_step, tf.int32)
43 warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32)
44
45 global_steps_float = tf.cast(global_steps_int, tf.float32)
46 warmup_steps_float = tf.cast(warmup_steps_int, tf.float32)
47
48 warmup_percent_done = global_steps_float / warmup_steps_float
49 warmup_learning_rate = init_lr * warmup_percent_done
50
51 is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32)
52 learning_rate = (
53 (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate)
54
55 # It is recommended that you use this optimizer for fine tuning, since this
56 # is how the model was trained (note that the Adam m/v variables are NOT
57 # loaded from init_checkpoint.)
58 optimizer = AdamWeightDecayOptimizer(
59 learning_rate=learning_rate,
60 weight_decay_rate=0.01,
61 beta_1=0.9,
62 beta_2=0.999,
63 epsilon=1e-6,
64 exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
65
66 if use_tpu:
67 optimizer = tf.tpu.CrossShardOptimizer(optimizer)
68
69 tvars = tf.trainable_variables()
70 grads = tf.gradients(loss, tvars)
71
72 # This is how the model was pre-trained.
73 (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0)
74
75 train_op = optimizer.apply_gradients(
76 zip(grads, tvars), global_step=global_step)
77
78 # Normally the global step update is done inside of `apply_gradients`.
79 # However, `AdamWeightDecayOptimizer` doesn't do this. But if you use
80 # a different optimizer, you should probably take this line out.
81 new_global_step = global_step + 1

Callers

nothing calls this directly

Calls 3

constantMethod · 0.80
apply_gradientsMethod · 0.45

Tested by

no test coverage detected

Used in the wild real call sites across dependent graphs

searching dependent graphs…