Synthesizer for Training

Implements Warmup learning rate schedule until 'warmup_steps', going from 'init_lr' to 'peak_lr' for multiple optimizers.

r"""Adaptive Layer Normalization

r"""Adaptive Layer Normalization

Conv1d + GLU(Gated Linear Unit) with residual connection. For GLU refer to https://arxiv.org/abs/1612.08083 paper.

Residual Vector Quantizer. Args: dimension (int): Dimension of the codebooks. n_q (int): Number of residual vector quantizers used

Implements 'Scaled Adam', a variant of Adam where we scale each parameter's update proportional to the norm of that parameter; and also lea

dataset class for text tokens to semantic model training.

Modifies the backpropped derivatives of a function to try to encourage, for each channel, that it is positive at least a proportion `threshol

Dialted and Depth-Separable Convolution

r""" sort the dataset wrt. input length divide samples into buckets sort within buckets divide buckets into batches sort batches

Maintain similar input lengths in a batch. Length groups are specified by boundaries. Ex) boundaries = [b1, b2, b3] -> any batch is inclu

Codebook with Euclidean distance. Args: dim (int): Dimension. codebook_size (int): Codebook size. kmeans_init (bool): Whet

MelStyleEncoder

Multi-Head Attention module

r"""Allows the model to jointly attend to information from different representation subspaces as described in the paper: `Attention Is All You

Residual vector quantization implementation. Follows Algorithm 1. in https://arxiv.org/pdf/2107.03312.pdf

Scaled Dot-Product Attention

Synthesizer for Training

Zero-pads model inputs and targets

1) loads audio, speaker_id, text pairs 2) normalizes text and converts them to sequences of integers 3) computes spectrograms from audio

r"""TransformerEncoder is a stack of N encoder layers. Users can build the BERT(https://arxiv.org/abs/1810.04805) model with corresponding paramet

r"""TransformerEncoder is a stack of N encoder layers. Users can build the BERT(https://arxiv.org/abs/1810.04805) model with corresponding paramet

Vector quantization implementation. Currently supports only euclidean distance. Args: dim (int): Dimension codebook_size (int)

This class adds to class Optimizer the capability to optimize parameters in batches: it will stack the parameters and their grads for you so

double_swish(x) = x * torch.sigmoid(x-1) This is a definition, originally motivated by its close numerical similarity to swish(swish(x)