Base class of Transfomer decoder module. Args: vocab_size: output dim encoder_output_size: dimension of attention attention_heads: the number of heads of multi head attention linear_units: the hidden units number of position-wise feedforward num_blocks: th
| 34 | |
| 35 | |
| 36 | class TransformerDecoder(torch.nn.Module): |
| 37 | """Base class of Transfomer decoder module. |
| 38 | Args: |
| 39 | vocab_size: output dim |
| 40 | encoder_output_size: dimension of attention |
| 41 | attention_heads: the number of heads of multi head attention |
| 42 | linear_units: the hidden units number of position-wise feedforward |
| 43 | num_blocks: the number of decoder blocks |
| 44 | dropout_rate: dropout rate |
| 45 | self_attention_dropout_rate: dropout rate for attention |
| 46 | input_layer: input layer type |
| 47 | use_output_layer: whether to use output layer |
| 48 | pos_enc_class: PositionalEncoding or ScaledPositionalEncoding |
| 49 | normalize_before: |
| 50 | True: use layer_norm before each sub-block of a layer. |
| 51 | False: use layer_norm after each sub-block of a layer. |
| 52 | src_attention: if false, encoder-decoder cross attention is not |
| 53 | applied, such as CIF model |
| 54 | query_bias: whether use bias in attention.linear_q |
| 55 | key_bias: whether use bias in attention.linear_k, False for whisper models. |
| 56 | value_bias: whether use bias in attention.linear_v |
| 57 | gradient_checkpointing: rerunning a forward-pass segment for each |
| 58 | checkpointed segment during backward. |
| 59 | tie_word_embedding: Tie or clone module weights depending of whether we are |
| 60 | using TorchScript or not |
| 61 | """ |
| 62 | |
| 63 | def __init__( |
| 64 | self, |
| 65 | vocab_size: int, |
| 66 | encoder_output_size: int, |
| 67 | attention_heads: int = 4, |
| 68 | linear_units: int = 2048, |
| 69 | num_blocks: int = 6, |
| 70 | dropout_rate: float = 0.1, |
| 71 | positional_dropout_rate: float = 0.1, |
| 72 | self_attention_dropout_rate: float = 0.0, |
| 73 | src_attention_dropout_rate: float = 0.0, |
| 74 | input_layer: str = "embed", |
| 75 | use_output_layer: bool = True, |
| 76 | normalize_before: bool = True, |
| 77 | src_attention: bool = True, |
| 78 | query_bias: bool = True, |
| 79 | key_bias: bool = True, |
| 80 | value_bias: bool = True, |
| 81 | activation_type: str = "relu", |
| 82 | gradient_checkpointing: bool = False, |
| 83 | tie_word_embedding: bool = False, |
| 84 | use_sdpa: bool = False, |
| 85 | layer_norm_type: str = 'layer_norm', |
| 86 | norm_eps: float = 1e-5, |
| 87 | n_kv_head: Optional[int] = None, |
| 88 | head_dim: Optional[int] = None, |
| 89 | mlp_type: str = 'position_wise_feed_forward', |
| 90 | mlp_bias: bool = True, |
| 91 | n_expert: int = 8, |
| 92 | n_expert_activated: int = 2, |
| 93 | ): |