Define a hyperparameter. random_val_fn takes a RandomState and returns a random hyperparameter value.

map self onto fn. If fn takes multiple args, tuple-unpacking is applied.

Group elements by selector and return a list of (group, group_records) tuples.

like [x for x in y if x.selector == value]

Given a list of datapoints, return a string describing their mean and standard error

If selector is a function, return selector. Otherwise, return a function corresponding to the selector string. Examples of valid selector

Pretty-print a 2D array of data, optionally with row/col labels

Global registry of hyperparams. Each entry is a (default, random) tuple. New algorithms / networks / etc. should add entries here.

Given records with a common test env, get the test records (i.e. the records with *only* that single test env and no other test envs)

Given all records from a single (dataset, algorithm, test env) pair, return a sorted list of (run_acc, records) tuples.

Given all records from a single (dataset, algorithm, test env) pair, return the mean test acc of the k runs with the top val accs.

Override the default train() to freeze the BN parameters

Auto-select an appropriate featurizer for the given input shape.

Unsafe, be careful you know what you're doing.

Return the {val_acc, test_acc} for a group of records corresponding to a single step.

For a dataset with n >= 3 envs, return all combinations of 1 and 2 test envs.

Return the dataset class with the given name.

Here a mask with continuous values in the range [0,1] is formed to control the amount of update for each parameter based on the agree

Given all records, print a results table for each dataset.

Given records from a run, return a {val_acc, test_acc} dict representing the best val-acc and corresponding test-acc for that run.