Layers of encoder transformer blocks plus an final layernorm. Args: num_layers (int): number of layers. dim_model (int): main dim

RMS LayerNorm

RMS LayerNorm

Open dataset in readonly mode. `DistributeDataset` is used to read datasets in a distributed manner. Data in this dataset will be distributed

The whole feed-forward block. A sequence of operation. Consists of layernorm, feed-forward and residual connection. Args: dim_model (int)

The whole feed-forward block. A sequence of operation. Consists of layernorm, feed-forward and residual connection. Args: dim_model (int)

r"""FeedForward module Args: dim_in (int): input dimension. dim_ff (int): middle dimension. dim_out (int, optional): outp

r"""FeedForward module Args: dim_in (int): input dimension. dim_ff (int): middle dimension. dim_out (int, optional): outp

The whole cross-attention block. A sequence of operation. Consists of layernorm, self-attention and residual connection. Args: dim_model

The whole cross-attention block. A sequence of operation. Consists of layernorm, self-attention and residual connection. Args: dim_model

The whole transformer block. A sequence of operation. Consists of self-attention block[, cross-attention block] and feed-forward block. Args:

The whole transformer block. A sequence of operation. Consists of self-attention block[, cross-attention block] and feed-forward block. Args:

model configuration

Layers of encoder transformer blocks plus an final layernorm. Args: num_layers (int): number of layers. dim_model (int): main dim