Pass the given input to the LLM to either generate a new output or just update the key value cache If the input is a string, the toke

Initialize the encoder, decoder and joint model of the Streaming STT FastConformer. Args: vocabulary_path: Valid file path to a .n

If the current autocast context is float16, cast it to bfloat16 if available (unless we're in jit) or float32

Temporarily change the working directory to the given path and change it back at the end Args: destination: global or local path

Transforms query, key and value. Args: query (torch.Tensor): (batch, time1, size) key (torch.Tensor): (batch, time2, s

Stateful prediction of scores and state for a (possibly null) tokenset. This method takes various cases into consideration :

Convert into overlapping chunks. Args: x (torch.Tensor): # (batch x head, time, size) w (int): Chunk overlap size

Calculates the output length of a Tensor passed through a convolution or max pooling layer

Resolve the given sequence of tokens to a text string Args: tokens: sequence of tokens Returns: string contain

Compute attention context vector. Args: value (torch.Tensor): (batch, time2, size) scores(torch.Tensor): (batch, time1

Sets maximum input length. Pre-calculates internal seq_range mask.

Same as sliding_chunks_matmul_qk but for prob and value tensors. Args: prob (torch.Tensor): (batch, head, time, size)

Matrix multiplication of query x key tensors using with a sliding window attention pattern. This implementation splits the input into overlapp

Maps the decoding state to the given device Args: dec_state: hidden state of the decoder model device: target device

Returns an LSTM with forget gate bias init to `forget_gate_bias`. Args: input_size: See `torch.nn.LSTM`. hidden_size:

Compute the attention output with global indices. Args: value (torch.Tensor): (batch, head, time, head_dim) The value ve

Compute the attention output of global tokens attending to all. Args: query (torch.Tensor): (batch, head, time, head_dim

Compute global attention indices. Args: is_index_global_attn (torch.Tensor): (batch, time) A boolean tensor indicating i

Prepare the trainable modules of the Joint Network Args: num_classes: Number of output classes (vocab size) excluding th

Prepare the trainable parameters of the Prediction Network. Args: vocab_size: Vocab size (excluding the blank token).

Convert diagonals into columns (or columns into diagonals depending on `direction` Args: x (torch.Tensor): (batch x head, chunk_c

Shift every row 1 step to right converting columns into diagonals Args: x (torch.Tensor): (batch x head, chunks_count + 1, w, 2w

Pass the given audio signal to the STT to get a transcription Args: processed_signal: preprocessed audio chunk process

Performs channel chunked convolution

Tries to split input by batch, run conv and concat results

For dw convs, tries to split input by time, run conv and concat results

Perform a single decoding step Args: label: previous predicted token hidden: hidden state after the last decoding ste

Reset and extend the positional encodings if needed.

Find supported sample rate and number of channels for a given audio device Args: audio: pyaudio instance device_inde

Flush Cuda cache to prevent side effect and slowdowns

Syntehsize the entire text Args: text: String of text that should be synthesized Returns: synthesized audio d

Instantiates object using DictConfig-based configuration

Perform decoder and joint step for every encoded signal in the given audio chunk Args: encoder_output: encoded audio chunk contain

Check stopping conditions and decide if the current sentence or sequence should be returned Returns: end: Boolean value if sentenc

Handle new transcription and decide if it should be passed to the LLM Args: transcribed_text: Transcription output from the STT. C

Initialize the state of the RNN layers, with same dtype and device as input `y`. Args: y: A torch.Tensor whose device th

Compute the joint step of the network after projection. Here, B = Batch size T = Acoustic model timesteps U

Perform a single joint step Args: enc: encoded audio signal pred: decoder output Returns: probabil

Collates the label inputs for the rnn-t prediction network. If `labels` is already in torch.Tensor form this is a no-op. Args: labels

Print our all available audio devices and their index

Start the recording process in the main thread and all other processes in a separate process.

Mask locations invalid for the sliding window attention Args: input_tensor (torch.Tensor): # (batch x head, time, size)

Record an audio stream from the microphone in a separate process Args: audio_buffer: multiprocessing queue to store the recorded

Resets token buffer after end of sentence

Resets current sentence and current sequence after end of sequence. Helper variables are set to start a new sequence when the next word is tra

This function sets the needed values and parameters to perform streaming. The configuration would be stored in self.streaming_cfg.

Resolve the given text into a sequence of tokens Args: text: string of arbitrary length Returns: Sequence of t

Translate a sequence of predicted tokens to text Args: prediction: sequence of tokens Returns: A string contai

Initialized tokenizer and LLM model from huggingface Args: model_name: huggingface model descriptor Returns: I

Initialize STT model and perform warm up steps Args: Returns: Instance of TTS model

Initialize STT, LLM and TTS model and the state of the voice assistant Args: Returns: Instance of voice assistant model

Construct an MultiHeadedAttention object.

Construct an RelPositionMultiHeadedAttention object.

Construct an RelPositionMultiHeadAttentionLongformer object.

Construct an PositionalEncoding object.

Initialize encoder, decoder and joint from given config