num_heads_kv: can be used to toggle MQA / GQA. If None, use num_heads. return_residual: whether to return the input x along with the

Copied from Mlp above. If num_layers > 2, add more Mlp layers, doubling each time.

For prenorm=True, this Block has a slightly different structure compared to a regular prenorm Transformer block. The standard

interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style).

If max_position_embeddings <= 0, there's no position embeddings If word_embe_proj_dim is not None (e.g., OPT-350m), we embed to that

Args: code_dir: the code directory use_git: if using git, then upload all files that are not ignored by git.

If model_cfg is passed, it will take precedence over cfg.model

Arguments: ignore_index: int. If labels == ignore_index, the loss is set to 0.0. label_smoothing: float l