Per token quantization to 8bits. No grouping is needed for quantization Args: x (Tensor): _description_ Returns: _type_: _description_
(x: Tensor)
| 12 | |
| 13 | |
| 14 | def activation_quant(x: Tensor): |
| 15 | """Per token quantization to 8bits. No grouping is needed for quantization |
| 16 | |
| 17 | Args: |
| 18 | x (Tensor): _description_ |
| 19 | |
| 20 | Returns: |
| 21 | _type_: _description_ |
| 22 | """ |
| 23 | scale = 127.0 / x.abs().max(dim=-1, keepdim=True).values.clamp_(min=1e-5) |
| 24 | y = (x * scale).round().clamp_(-128, 127) / scale |
| 25 | return y |
| 26 | |
| 27 | |
| 28 | class BitLora(BitLinear): |