Like logger.log, but also replaces all TensorFlow ops/tensors with their names. Sensitive to value of DEBUG_LOGGING, see enable_debug/disable_debug

Creates one layer of blocks for the ResNet model. Args: inputs: A tensor of size [batch, channels, height_in, width_in] or [batch, height

utility for retrieving polyak averaged params

Strided 2-D convolution with explicit padding. The padding is consistent and is based only on `kernel_size`, not on the dimensions of `inputs` (a

Performs a batch normalization followed by a ReLU.

Creates chain of length length. First node is constant, rest are tanh. Returns list of nodes. Advantage over make_chain_tanh is that unlike Variab

Creates resnet-like chain of length length. First node is constant, rest are tanh. Returns list of nodes. Has length - 2 articulation points (ie

sort list by op.name.

Evaluates gradient, prints peak memory in MBs.

Evaluates gradient, prints peak memory in MBs.

Creates forward backward graph using tf.gradients. A0->A1->A2->..->An / / / B0<-B1<-B2<-..<-Bn

utlity for keeping track of layer names

Authors: Tim Salimans & Yaroslav Bulatov memory efficient gradient implementation inspired by "Training Deep Nets with Sublinear Memory Cost

Creates loss tensor for resnet model.

Create session with optimizations disabled.

Creates chain of length length. First node is Variable, rest are tanh. Returns nodes. Note, if length is 1, there are no non-linearities in the gr

Helper method to use instead of sess.run that will automatically capture run_metadata.

Decorate tf.gradients calls with explicit device placement to avoid memory leaks when splitting model across multiple GPUs

Helper utility to help variation in targets input type: -- it can be iterable vs simple list -- it can contain Operation objects or Tensor objects

/job:localhost/replica:0/task:0/device:CPU:0 -> /cpu:0

Return sorted memory allocation records from list of nodestats [NodeExecStats, NodeExecStats...], it's the run_metadata.step_stats.dev_stats[0].st

convolutional layer

transposed convolutional layer

Helper method for printing ops. Converts Tensor/Operation op to op.name, rest to str(op).

Helper method for printing ops. Converts Tensor/Operation op to op.name, rest to str(op).

Evaluates gradient, prints peak memory.

Convert graph nodes to hashes (networkx is_isomorphic needs integer nodes).

Obtain a single valid execution order which approximately minimizes peak memory usage. TODO: deprecate/hide modify_graph arg Args: targets:

numerically stable log_sum_exp implementation that prevents overflow

Evaluates gradient, returns memory in MB's and gradient eval time in seconds.

Evaluates gradient, returns memory in MB's and gradient eval time in seconds.

Sort nodes by estimated memory usage.

Reverses direction of all edges in the graph.

Runs optimization for few steps, returns loss.

Like test_chain, but use automatic rewriting with checkpoints="memory" strat.

Like test_chain, but use automatic rewriting with checkpoints="memory" strategy.

Creates loss tensor for resnet model.

Returns dictionary of device_name->[NodeExecStats, NodeExecStats...]

Length is number of concats.

Given a list of NodeExecStats messages, construct memory timeline.

Return index of largest entry

pretty print parsed memory timeline.

Force operation a to run after b. Do not add control dependencies to ops that already run after. Returns 0 if no dependencies were added, 1 otherw

Sorts nodes according to order provided. Args: nodes: nodes to sort total_order: list of nodes in correct order dedup: if True, also

Returns children, counting control outputs as children.

Return parts of the graph needed to compute targets.

Creates loss tensor for resnet model.

Creates loss tensor for resnet model.

Creates loss tensor for resnet model.

Creates loss tensor for resnet model.