Class LayerwiseEqualizationPass

ppq/quantization/optim/equalization.py:23–376

## Layer-wise Equalization Pass(层间权重均衡过程) Weight distributions can differ strongly between output channels, using only one quantization scale, per-tensor quantization has its trouble for representing the value among channels. For example, in the case where one channel has weights

class LayerwiseEqualizationPass(QuantizationOptimizationPass):
    """
    ## Layer-wise Equalization Pass(层间权重均衡过程)

    Weight distributions can differ strongly between output channels, 
    using only one quantization scale, per-tensor quantization has its trouble for representing the value among channels.

    For example, in the case where one channel has weights in the range [−128, 128] and another channel has weights in the range (−0.5, 0.5), 
    the weights in the latter channel will all be quantized to 0 when quantizing to 8-bits.

    Hopefully, the performance can be improved by adjusting the weights for each output channel such that their ranges are more similar.
    
    Formula:
    
            Take 2 convolution layers as an example
            
            Where Y = W_2 * (W_1 * X + b_1) + b_2
            
            Adjusting W_1, W_2 by a scale factor s:
            
            Y = W_2 / s * (W_1 * s * X + b_1 * s) + b_2

            Where s has the same dimension as the output channel of W_1

    This method is called as Layer-wise Equalization, which is proposed by Markus Nagel.
    
    https://openaccess.thecvf.com/content_ICCV_2019/papers/Nagel_Data-Free_Quantization_Through_Weight_Equalization_and_Bias_Correction_ICCV_2019_paper.pdf

    self, iterations: int, weight_threshold: float = 0.5,
    including_bias: bool = False, including_activation: bool = False,
    bias_multiplier: float = 0.5, activation_mutiplier: float = 0.5,
    interested_layers: List[str] = None, optimize_level: int = 2,
    verbose:bool = False
    
    ### Parameters:

    * iterations(int):
            
            Integer value of Algorithm iterations.
            
            More iterations will give more plainness in your weight distribution,
            iteration like 100 can flatten all the parameter in your network to a same level.
            
            You are not recommended to iterate until value converges, 
            in some cases stop iteration earlier will give you a better performance.
            
    * weight_threshold(float)

            A threshold that stops processing value that is too small.

            By default, the scale factor of equalization method is computed as sqrt(max(abs(W_1)) / max(abs(W_2))),
            the maximum value of W_2 can be very small(like 1e-14), while the maximum value W_1 can be 0.5.

            In this case, the computed scale factor is 1e7, the optimization will loss its numerical stability and even give an unreasonable result.

            To prevent the scale factor becoming too large, ppq clips all the value smaller than this threshold before iterations.
            
            This parameter will significantly affects the optimization result.