Cycles through alternately training the shared parameters and the controller, as described in Section 2.2, Training ENAS and Deriving

Samples a set of `args.num_blocks` many computational nodes from the controller, where each node is made up of an activation function, and

Tokenizes a text file.

Drops out weights to implement DropConnect. Args: w_raw: Full, pre-dropout, weights to be dropped out. dropout_p: Proportion of w

Returns a the `torch.no_grad` context manager for PyTorch version >= 0.4, or a no-op context manager otherwise.

TODO(brendan): We are always deriving based on the very first batch of validation data? This seems wrong...

Computes the loss for the same batch for M models. This amounts to an estimate of the loss, which is turned into an estimate for the

Computes the perplexity of a single sampled model on a minibatch of validation data.

Fixes the shared parameters and updates the controller parameters. The controller is updated with a score function gradient estimator

Train the language model for 400 steps of minibatches of 64 examples. Args: max_step: Used to run extra training steps as

Based on `args`, optionally adds regularization penalty terms for activation regularization, temporal activation regularization and/or hidden

Checks `model` for a new largest gradient for this epoch, in order to track gradient explosions.

Constructs a set of DAGs based on the actions, i.e., previous nodes and activation functions, sampled from the controller/policy pi. Args:

Logs the controller's progress for this training epoch.

Logs a set of training steps.

Creates and initializes the shared and controller models.

Computes a single pass through the discovered RNN cell.

Evaluate on the validation set. NOTE(brendan): We should not be using the test set to develop the algorithm (basic machine learning g

main: Entry point.

Constructor for training algorithm. Args: args: From command line, picked up by `argparse`. dataset: Currently only `

Embedding constructor. Args: dropout: Dropout probability. scale: Used to scale parameters of embedding weight matrix

Embeds `inputs` with the dropped out embedding weight matrix.

Parses all of the arguments above, which mostly correspond to the hyperparameters mentioned in the paper.

Sets the directories for the model, and creates those directories. Args: args: Parsed from `argparse` in the `config` module.

Converts x, possibly scalar and possibly tensor, to a Python scalar.