Power function. \param base first argument \param exp second argument \return \a base raised to \a exp template<typename T,typename U> typename enable

Prepare the internal variables that would used in the monkey patching mechanism later. 1. Set up the list of quantized modules that a

Execute inference on a batch of images. The images should already be batched and preprocessed, as prepared by the ImageBatcher class.

Registers a callback with the logger, which will be invoked when the logging severity is modified. The callback is guaranteed to be c

Reset this calibrator for reuse. The calibrator will clear any dynamic ranges cached from previous calibration runs, and wil

You should always call this function after a use case because T5TRTDecoder does not clear the cached encoder_hidden_states or cross_attention

r"""ResNet-18 model from `"Deep Residual Learning for Image Recognition" <https://arxiv.org/pdf/1512.03385.pdf>`_. Args: pretrained (

Reverse the effect of monkey patch by using the orginal_func_map to replace back the original modules.

Save the ONNX model to the given location. :param output_path: Path pointing to the location where to write out the updated ONNX mode

Multiply by power of two. \param arg number to modify \param exp power of two to multiply with \return \a arg multplied by 2 raised to \a exp template

Multiply by power of two. \param arg number to modify \param exp power of two to multiply with \return \a arg multplied by 2 raised to \a exp template

\brief Set the ErrorRecorder for this interface Assigns the ErrorRecorder to this interface. The ErrorRecorder will track all errors during execution

\brief Set Int8 Calibration interface. The calibrator is to minimize the information loss during the INT8 quantization process.

\brief Set the kernel weights for the convolution. The weights are specified as a contiguous array in \p GKCRS order, where \p G is the number of gro

\brief Set the memory size for the memory pool. TensorRT layers access different memory pools depending on the operation. This function sets in the I

\brief Set the number of groups for a convolution. The input tensor channels are divided into \p nbGroups groups, and a convolution is executed for

Set the types and shapes dict for kv-cache based on the provided inputs: kv_dict: Dict[str, tuple/torch.dtype], the dict to modify within

Sets the return device of the return via to(). Device name should be the same as torch devices: cuda, cpu, etc. This is used in our m

Indicate that ``iterate()`` should stop. This may be invoked from any of the callbacks provided to ``iterate()``.

\brief Construct an empty Dims2 object.

Returns a class (not an instance) that will provide a list-like interface with runtime type checks. Note: The types are provided lazily

Args: implementation (int): The implementation for this Algorithm. tactic (int):

Scan the input string and extract the first float instance.

Compute learning rate at given step.

Helper for child classes to iterate through a slice of data. Return: (Union[Dict[str, str], List[str]]): Returns a list

Shared function body between TensorQuantFunction and FakeTensorQuantFunction

\brief Add a concatenation layer to the network. \param inputs The input tensors to the layer. \param nbInputs The number of input tensors. \see ICo

\brief Make an output for this loop, based on the given tensor. axis is the axis for concatenation (if using outputKind of kCONCATENATE or kREVERSE).

\brief Add a MatrixMultiply layer to the network. \param input0 The first input tensor (commonly A). \param op0 The operation to apply to input0. \pa

\brief Add a plugin layer to the network using the IPluginV2 interface. \param inputs The input tensors to the layer. \param nbInputs The number of i

\brief Add a shuffle layer to the network. \param input The input tensor to the layer. \see IShuffleLayer \return The new shuffle layer, or nullptr

Add args needed for perf benchmarking mode.

This method parses object of QuantizationSpec class and fill in `layerwise_config` dictionary holding information about layers that n

Network with VGG like architecture.

\brief Creates the network, configures the builder and creates the network engine. \details This function creates the MNIST network by parsing the ON

Args: run_results (RunResults): The result of Comparator.run() fail_fast (bool): Whether to exit after the first fa

Confirms whether or not user wants to delete given folder path. Args: fpath (str): Path to folder. prompt (str): Prompt to d

1x1 convolution

1x1 convolution

3x3 convolution with padding

3x3 convolution with padding

Converts `text` to Unicode (if it's not already), assuming utf-8 input.

\brief Copy the contents of input host buffers to input device buffers synchronously.

\brief Copy the contents of input host buffers to input device buffers asynchronously.

\brief Copy the contents of output device buffers to output host buffers synchronously.

\brief Copy the contents of output device buffers to output host buffers asynchronously.