Dataclass to hold multimodal ChatML data.

We implement a dual-path FFN decoder layer where the audio tokens and text tokens go through separate FFN layers. The audio and text tokens share

Sample collator for Higgs-Audio model. Args: whisper_processor (WhisperProcessor): The whisper processor. audio_in_token_id (int)

Represents a chunk of generated content, either text or audio tokens.

Introduced in SoundStream: An end2end neural audio codec https://arxiv.org/abs/2107.03312

ArithmeticCoder, Let us take a distribution `p` over `N` symbols, and assume we have a stream of random variables `s_t` sampled from `p`. Let

ArithmeticDecoder, see `ArithmeticCoder` for a detailed explanation. Note that this must be called with **exactly** the same parameters and seque

Async streamer that handles both text and audio token generation from Higgs-Audio model. Stores chunks in a queue to be consumed by downstrea

1x1 Conv1d.

Decoder block (no up-sampling)

Codebook with Euclidean distance. Args: dim (int): Dimension. codebook_size (int): Codebook size. kmeans_init (bool): Whet

Codebook with Euclidean distance. Args: dim (int): Dimension. codebook_size (int): Codebook size. kmeans_init (bool): Whet

Projection layers that map hidden states from the LLM component to audio / text logits. We support two type of audio head: - Basic Audio Head

Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`WhisperEncoderLayer`]. Args:

Configuration of the Audio encoder in Higgs-Audio.

Projector that maps audio features extracted by Whisper to hidden state of the text model.

Outputs of HiggsAudio generation models, when using non-beam methods. Args: sequences (`torch.LongTensor` of shape `(batch_size, seq

Split an existing `nn.Embedding` module that splits the embedding into: - A frozen embedding for indices [0..freeze_until_idx]. - A trainable

A wrapper around nn.Linear to partially freeze part of the weight matrix.

Residual vector quantization implementation. Follows Algorithm 1. in https://arxiv.org/pdf/2107.03312.pdf

Residual Vector Quantizer. Args: dimension (int): Dimension of the codebooks. n_q (int): Number of residual vector quantizers used

Vector quantization implementation. Currently supports only euclidean distance. Args: dim (int): Dimension codebook_size (int)

Vector quantization implementation. Currently supports only euclidean distance. Args: dim (int): Dimension codebook_size (int)

Implementation of VQ similar to Karpathy's repo: https://github.com/karpathy/deep-vector-quantization Additionally uses following tricks

Stopping criteria that checks for stop signal from a threading event. Args: stop_signal (threading.Event): Event that will receive s

Override the forward call in lightning so it goes to training and validation step respectively

Enum for different generation modes in HiggsAudio model.

r""" This is the configuration class for the HiggsAudioModel. Args: text_config (`Union[AutoConfig, dict]`): The config o

Higgs-Audio is an end-to-end multimodal model with the capability to understand and generate text / audio. Consider the following example for mix

Residual vector quantization implementation. Follows Algorithm 1. in https://arxiv.org/pdf/2107.03312.pdf

Divide tokens equally among the sequence parallel ranks

All gather tensors among the ranks.

Gather tokens among the sequence parallel ranks