Checks whether callable accepts given keyword argument Parameters ---------- function: callable Function which which arguments

Create test directory and test data

Helper method for setting activation functions

Helper method that builds preprocesing layer for UCI Census Income dataset.

Builds soft parameter sharing multi-task model n_features: int Number of features in the data l1_regularizer: float L1 pe

Create Multigate Mixture of Experts Model Parameters ---------- input_shape: int Number of features apply_dropout: bool

Create Multigate Mixture of Experts Model Parameters ---------- input_shape: int Number of features apply_dropout: bool

Helper method to build task specific networks

Create Mixture of Linear Model Experts Parameters ---------- input_shape: int Number of features apply_dropout: bool

Regularize l2/l1 distance between variables Parameters ---------- variables: List of lists List of variables norm: str

Builds Mixture of Experts Layers if they are not provided by the user

Preprocess categorical column: use either embeddings or one-hot. Categorical feature can be represented either as an embedding or as

Split feature from targets, remove education, income_50K and marital_stat from the features as described in section 6.3

Builds main component of the layer by creating dropout layer that has the same binary mask for each element in the batch. Pa

Helper method to build expert networks for OMOE and MMOE

Config of the MOE layer

Creates model with cross-stitch units Parameters ---------- n_features: int Shape of the input

Creates Mixture of Experts Model

Checks whether data are in path directory or in current directory if path is not specified, loads data from the uci and splits test set into

Encode categorical features, represent them as dense set of integers and save the mapping from c. Normalize all of numeric feature using min-max

Checks Goodness of fit for single task problem

Builds Mixture of Experts Layers if they are not provided by the user

Build model for Cross Stitched networks Parameters ---------- hp: instance of HyperParameters Hyper-Parameters that define a

Build model for L2 constrained multi-task learning model Parameters ---------- hp: instance of HyperParameters Hyper-Paramet

Build hypermodel either for Multi-Gate Mixture of Experts or One-Gate Mixture of experts Parameters ---------- hp: instance of H

Build model for L2 constrained multi-task learning model Parameters ---------- hp: instance of HyperParameters Hyper-Paramet

Defines forward pass of the Preprocessing Layer, Builds embeddings for sparse categorical features and one-hot encodings for

Forward pass of the Multi-Gate Mixture of Experts model. Parameters ---------- inputs: np.array or tf.Tensor

Defines forward pass for the layer, by dropping experts and then renormalizing utilization probabilities for remaining experts.

Defines set of computations performed in the MOE layer. MOE layer can accept single tensor (in this case it assumes the same

Forward pass of the MLP Parameters ---------- inputs: np.array or tf.Tensor Input to the model Re

Forward pass through constraining layer. Constraining layer can accept single tensor (in this case it assumes the same input

Forward pass through cross-stitch block Parameters ---------- inputs: np.array or tf.Tensor List of task

Forward pass of the One-Gate Mixture of Experts Model Parameters ---------- inputs: np.array or tf.Tensor

Get configuration for Preprocessing Layer

Get configuration of the Multi-Gate Mixture of Experts

Config of the ExpertUtilizationDropout layer

Get MLP configuration

Get group two data as described in section 6.3 of the paper. Second group of data contains two binary target variables: 1] Ma

Get group one data as described in section 6.3 of the paper. First group of data contains two binary target variables: 1] Mar

Get raw data

Create models with Mixture of Experts Layer, one model with dropout and another one without dropout.

Create intances of One-Gate Mixture of Experts Model with / without expert dropout and with / without base layer

Create MMOEs models for testing

Evaluates set of hyperparameters Parameters ---------- trial: kerastuner.Trial Object that holds informa

Test that elements of each column are convex combinations, and that output is a list of tensors with the same dimensionality

Checks how well model fitted training data, we want to check that model can perfectly fit data (i.e. overfit)

Checks whether layer can properly handle two types of input: 1) list of tensors 2) tensors And whether output shape

Tests saving and loading

Checks saving and loading functionality of multitask model