Class LearnedStepSizePass

ppq/quantization/optim/training.py:569–863

## Learned Step Size Pass(网络微调过程-LSQ) Learned Step Size optimization, a training-based optimization pass that tunes weights and scales for high precision quantization. [This method is proposed by Steven K. Esser] (https://arxiv.org/pdf/1902.08153.pdf) This is an alternative versi

class LearnedStepSizePass(TrainingBasedPass):
    """
    ## Learned Step Size Pass(网络微调过程-LSQ)

    Learned Step Size optimization, a training-based optimization pass that tunes weights and scales for high precision quantization.

    [This method is proposed by Steven K. Esser] (https://arxiv.org/pdf/1902.08153.pdf)

    This is an alternative version of LSQ, this pass will split your graph into multiple trainable blocks, each blocks will be trained separately.
    Warning: PPQ Learned Step Size minimize only the output loss of each block, which means after training the internal results probably goes far away from original. 

    PPQ Learned Step Size optimization requires 256 ~ 2048 samples for finetuning your network, while the data label is not necessary. All training data are cache in GPU memory or CPU memory for acceleration.

    The training loss will be computed as:

        let: Y = WX + b

        Quant(Y, scale_Y) = Quant(W, scale_W) Quant(X, scale_X) + b

        loss = loss_func(Y, Quant(Y, scale_Y)) # loss between fp output and int8 output, that is why we do not need labeled data.

    The formula of calculating the derivatives of y and scale_Y:

        if y > scale_Y * -128 and y < scale_Y * 127:
        
        dQuant(y, scale_Y)/dy       = dQuant(y, scale_Y)
        
        dQuant(y, scale_Y)/dscale_Y = Quant(y, scale_Y) - y

        if y < scale_Y * -128:
        
        dQuant(y, scale_Y)/dy       = 0
        
        dQuant(y, scale_Y)/dscale_Y = -128

        if y > scale_Y * 127:
        
        dQuant(y, scale_Y)/dy       = 0
        
        dQuant(y, scale_Y)/dscale_Y = 127

    ### Parameters:

    * interested_layers(List[str]):

            A list of operation names, only the layers listed in this parameter will be trained.

            If interested_layers is None, all layers(conv and gemm) will be trained.

    * steps(int)

            Training steps for finetuning your network, default is 500.

    * block_size(int)

            PPQ Learned Step Size optimization split your graph into blocks at first, 
            each block will be finetuned separately.