Per token quantization to 8bits. No grouping is needed for quantization Args: x (Tensor): _description_ Returns: _type_: _description_
(x: Tensor)
| 4 | |
| 5 | |
| 6 | def activation_quant(x: Tensor): |
| 7 | """Per token quantization to 8bits. No grouping is needed for quantization |
| 8 | |
| 9 | Args: |
| 10 | x (Tensor): _description_ |
| 11 | |
| 12 | Returns: |
| 13 | _type_: _description_ |
| 14 | """ |
| 15 | scale = 127.0 / x.abs().max(dim=-1, keepdim=True).values.clamp_(min=1e-5) |
| 16 | y = (x * scale).round().clamp_(-128, 127) / scale |
| 17 | return y |
| 18 | |
| 19 | |
| 20 | def weight_quant(w: Tensor): |