dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow sca

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

r""" Performs the in-projection step of the attention operation. This is simply a triple of linear projections, with shape constraints on the

r""" Performs the in-projection step of the attention operation, using packed weights. Output is a triple containing projection tensors for qu

Make causal mask used for bi-directional self-attention.

Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_l

Validate the `rope_scaling` configuration.

Replace consecutive `num_spk_embs` speaker embedding placeholders in input_embeds with pre-prepared speaker embeddings. This is an in-place repla

image_size: image_size or (image_height, image_width) return: pos_embed: [image_height, image_width, embed_dim]

r""" Extract full audio embeddings with optional chunk-based attention. This method computes embeddings for all audio frames at once,

r""" Extract audio embeddings in a streaming manner using cached key-value pairs. This method processes incoming audio features incre