A lightweight wrapper for Routing Transformer encoder stack. Args: model_dim: specifies dimension of transformer layers, token embeddings, and positional embeddings as well context vectors (attention values). hidden_dim: hidden dim of model. num_layers: number of transformer layers.
(model_dim,
hidden_dim,
num_layers,
num_heads,
left_context,
right_context,
block_size,
num_routing_layers,
num_routing_heads,
num_clusters,
attention_window,
atten_dropout_prob=0.,
relu_dropout_prob=0.,
residual_dropout_prob=0.)
source not stored for this graph (policy: none)