Should NOT be used externally. Return a new config used only for the tests.

Get valid sampling log. :return: get the valid sampling log output.

For some cases we don't need to insert QDQ patterns.

Takes one float input to an op, and converts it to quantized form.

Detect the rule matched nodes collections. Returns: [List] -- [the matched rule] [String] -- [the list contains the m

Detect the rule matched nodes collections. Returns: [List] -- [the matched rule] [String] -- [the list contains the m

Detect the rule matched nodes collections. Returns: [List] -- [the matched rule] [String] -- [the list contains the m

Check whether the identifier represents an operator type. Args: name (str): Operator identifier. Returns: bo

Create dictionary mapping list items to their indices in the list.

Setup the logger format and handler.

Create representative test inputs.

Matrix multiply matching JAX's accumulation behavior. JAX accumulates bfloat16 matmul in float32 and casts the result back to bfloat16. Repli

:param inps: torch.Tensor A torch tensor of shape [batch, (sequence_ctx + sequence_cont)] or of shape [batch, sequenc

Convert an operator type to a string key. Args: op_type (Callable): Operator type or callable object. Returns:

Parse ckpt batch norm inputs to match correct moving mean and variance. Args: graph_def (graph_def): original graph_def Returns:

Delete the temporarily files generated during the quantization process. :return: None

Unwrap all the modules from QDQ (Quantize-Dequantize) operations. Returns: None

Return numeric value whether variable has `.value` or is a plain array.

Set fp32 weights to qmodel.

Init quantizer.

Splits text to a list of string tokens.

Splits a token into subtokens defined in the subtoken dict.

Convert list of int subtoken ids to a list of string tokens.

Parse the neural_compressor wrapped configuration to Tensorflow. Args: tuning_cfg (dict): configuration for quantization.

Convert string from unicode to native format (required in Python 2).

Computes the accuracy over the k top predictions for the specified values of k

Computes the accuracy over the k top predictions for the specified values of k

Computes the accuracy over the k top predictions for the specified values of k

Computes the accuracy over the k top predictions for the specified values of k

Add inputs and outputs to gptq object.

Extend a dtype size function to support FP8 dtypes. Args: function (Callable): Function that returns the size of a dtype in bits. Re

Update scale registry

Adjust weight of LLM saved_model by scale.

Apply dequantize + batchmatmul/batchmatmulv2 + quantizev2 fusion. Dequantize + BatchMatMulV2 + QuantizeV2

Apply Conv3D Add Addn fusion. Dequantize + Conv3D + BiasAdd + Add + QuantizeV2 Dequantize + Conv3D + BiasAdd + AddV2 + QuantizeV2

Awq tensor convert tool. Reverse_reorder_int_tensor

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Dataset return data per batch.

Calibration model on hpu block-by-block to reduce device memory usage. Args: model (torch.nn.Module): prepared model. dataloader

Manually construct a QuantLinear since the class does not define an __init__. Steps: 1. Instantiate the module 2. Register parameter

Build Tensorflow saved model. Args: root (str, optional): path to saved model. Defaults to None. Returns: ro

Q-side of split QDQ: observe inputs during calibration. During calibration, runs the observer on inputs. After post_quantization_clea

Replace generate function for calibration and restore on demand. Args: self (keras.Model): Causal language model instance to modify.

Check the node has the const input. Args: node_name (string): node name Returns: bool: True if the node has

Clean files saved by each rank after gathering.

Configure the quantizer.

Rebuild function from graph_def. Args: func: The original concrete function get from saved_model. graph_def: The optimized graph

Converts a prepared model to a quantized model. Args: model (torch.nn.Module): The prepared model to be converted.

Convert the calibrated model into qdq mode. Args: model (GraphModule): The prepared model. Returns: GraphMod

Converts a prepared model to a quantized model. Args: model (QuantizationInterceptionModule): the prepared model to be convert

Loads a data file into a list of `InputBatch`s.

To convert INC fp8 model weight format to the vllm compatible format quantized by llm-compressor. Args: state_dict (dict): state_dict from

Create calibration data for static quantization.

r"""Decode a long from a two's complement little-endian binary string. >>> decode_long(b'') 0 >>> decode_long(b"\xff\x00") 255 >>

Setup default auto-tune settings. Returns: dict: A dictionary of operation-wise loss values with respect to alpha values.

Traverse the model definition and return the attention blocks and ffn blocks. Returns: blocks: A dict include the detected att

Bypass the module.

Execute BF16 convert. Returns: Transformed graph

Download hugging face model from hf hub.

dequantize and quantize weight with different scales.

This is a function to dump quantizable ops of model to user. Args: user_cfg (dict): quantization config Returns: None

Enable the module.

Makes sure that a tensor name has :0 if no explicit port exists.

Custom evaluate function to estimate the accuracy of the model. Args: model (tf.Graph or string or INC.model.TensorflowCheckpointModel):

Custom evaluate function to estimate the accuracy of the model. Args: model (tf.Graph): The input model graph. Returns: