Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/ab

Given input length, compute how many spans should be masked

params: mel: [T, num_mels], torch.tensor return: wav: [1, T], torch.tensor

Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3

Calculate the mel spectrogram of an input signal. This function uses slaney norm for the librosa mel filterbank (using librosa.filters.mel) a

semantic_tokens: [T_1], torch.LongTensor xt: [T_2, 80], torch.Tensor, DO NOT normalize it outside ode_steps: int, number of o

Arguments: timbre_speech: torch.Tensor, shape [B, N_speech_24k] timbre_semantic_token: torch.Tensor, shape [B, N]

mel_chunk: [T, 80], torch.Tensor, T is the chunk size semantic_tokens_chunk: [T], torch.LongTensor start_position_id: int, de

Initialization. INPUT: - in_features: shape of the input - alpha: trainable parameter alpha is in

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.

Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, th

Make causal mask used for bi-directional self-attention.

Make causal mask used for bi-directional self-attention.

Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779).

params: wav_path: str, path of the wav, should be 24k wav_data: torch.tensor or numpy array, shape [T], wav data, sho

hidden_states: [bsz, tgt_len, embed_dim] seq_lens: Optinal, [bsz]

messages: List[Dict] messages[i] = { "role": "user" | "assistant" | "system", "content": str }

Open an audio file and read as mono waveform, resampling as necessary Parameters ---------- file: str The audio file to open

Compute the log-Mel spectrogram of Parameters ---------- audio: Union[str, np.ndarray, torch.Tensor], shape = (*) The path t

Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored.

Make dataset and collator for supervised fine-tuning.

load the mel filterbank matrix for projecting STFT into a Mel spectrogram. Allows decoupling librosa dependency; saved using: np.sav

Pad or trim the audio array to N_SAMPLES, as expected by the encoder.

Collects the state dict and dump to disk.

Sample from audio logits with top-k, temperature and repetition penalty. Args: logits: Logits tensor of shape [batch_size, seq_le

Sample from text logits with top-k, temperature and repetition penalty. Args: logits: Logits tensor of shape [batch_size, seq_len

Implementation of sinc, i.e. sin(pi * x) / (pi * x) __Warning__: Different to julius.sinc, the input is multiplied by `pi`!

Create sinusoidal timestep embeddings. :param t: a 1-D Tensor of N indices, one per batch element. These ma