Custom linear layer with bit quantization. Args: dim (int): The input dimension of the layer. training (bool, optional): Whe

BitNetTransformer is a transformer-based model for BitNet. Args: dim (int): The dimension of the token embeddings. depth (in

BitFeedForward module performs feed-forward operations on the input tensor. Args: dim (int): The input dimension. dim_out (i

AutoregressiveWrapper is a wrapper class that adds autoregressive generation functionality to a given neural network. Args: net (nn.

Multi-head grouped query attention (GQA) layer. Reference: "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Chec

Transformer module that applies multi-head attention and feed-forward layers. Args: dim (int): The dimension of the input and output

Multi-head attention module.

Custom linear layer with bit quantization. Args: dim (int): The input dimension of the layer. training (bool, optional): Whe

BitMamba module for performing computations using the BitNet architecture. Args: dim (int): The input dimension (D). depth (

BitMoE (Bitwise Mixture of Experts) module. Args: dim (int): The input dimension. num_experts (int): The number of experts i

An MLP is a simple linear layer followed by a non-linearity i.e. each Expert Args: dim (int): The input dimension of the linear layer.

Gated Linear Unit (GLU) module. Args: dim_in (int): Input dimension. dim_out (int): Output dimension. activation (Ca

A class representing a Noisy Top-k Router module. This module takes the output tensor from a multihead self attention block and performs rou

OneBitViT is a vision transformer model for image classification tasks. Args: image_size (int or tuple): The size of the input image

BitLora class represents a custom linear layer with LoRa (Low Rank) regularization. Args: rank (int): The rank of the LoRa regulariz

A class used to perform inference with the BitNetTransformer model. ... Attributes ---------- model : torch.nn.Module a

The CachedWheelsCommand plugs into the default bdist wheel, which is ran by pip when it cannot find an existing wheel (which is currently the

Root Mean Square Normalization (RMSNorm) module. Args: dim (int): The input dimension. affine (bool, optional): If True, app