RTX Neural Texture Compression (NTC) SDK v0.10.0 BETA

Quick Start Guide

Introduction

Neural Texture Compression (NTC) is an algorithm designed to compress all PBR textures used for a single material together. It works best when the texture channels are correlated with each other, for example, detail in the albedo texture corresponds to detail in the normal texture. Up to 16 texture channels can be compressed into one NTC texture set. Typical PBR materials have 9-10 channels: 3x albedo, 3x normal, metalness, roughness, ambient occlusion, opacity.

During compression, the original texture data is transformed into a combination of weights for a small neural network (decoder) and a tensor of latents or features that are sampled and passed through the decoder to reconstruct the texture colors, as is illustrated below. The sampling and decoding processes are fast enough to use them directly in the shaders that normally sample the material textures, such as base pass pixel shaders or ray tracing hit shaders. However, the decoder produces unfiltered data for only one texel, and in order to get filtered textures, we suggest using NTC in combination with Stochastic Texture Filtering (STF). For renderers targeting lower-end hardware, we suggest implementing the "Inference on Load" mode where NTC textures are decompressed when the game or map is loaded, and transcoded to one of the block-compressed formats (BCn) at the same time. There is also an advanced "Inference on Feedback" mode that uses Sampler Feedback to find the set of texture tiles needed to render the current view and then decompresses only those tiles, storing them in a sparse tiled texture as BCn.

For more background information, please refer to the Random-Access Neural Compression of Material Textures page on the NVIDIA Research website.

Example Compression Rates

NTC can be thought of as an adjustable quality/constant bitrate lossy compression scheme. This means that it will attempt to reconstruct the input images with minimal error while using a fixed amount of data specified as a compression time target. However, unlike block compression schemes (which have a fixed data rate for a given format) the per-texel memory footprint of a NTC-compressed texture bundle will vary based in the specified Latent Shape (which is the composite of the number of high- and low-resolution latent channels, the bit-depth of those high- and low-resolution channels, and the scale factor between them). Each latent shape corresponds to a given per-texel bitrate, and thus for a desired compression level a compatible latent shape can be selected which corresponds to desired bitrate as closely as possible. Furthermore, although NTC compression requires specifying a latent shape (and thus bitrate) it is possible to approximate a constant quality/variable bitrate approach by performing pre-analysis of the bundle to determine what formats are required to achieve a target quality level. See Adaptive Compression for more.

To demonstrate how NTC compares to other methods, consider a material defined by the following bundle of material textures:

• Albedo Color (RGB)
• Normal (XY)
• Roughness
• Metalness
• Ambient Occlusion Factor

Example material from MetalPlates013 from AmbientCG

Assuming 8 bits per channel and optimal channel packing, this corresponds to a bitrate of 64 bits/texel. In a contemporary pipeline this might be block-compressed into one BC7 texture for Albedo (8 bits/texel), one BC5 texture for normals (8 bits/texel), and a third BC7 texture for Roughness, Metalness, and Ambient Occlusion packed as separate channels (for another 8 bits/texel). For NTC we have found that many real-world texture bundles of this format can be compressed with results comparable to BCn (a PSNR of 40 to 50 dB) with a latent shape requiring about 5 bits/texel.

If we assume a 2k-by-2k texture resolution (and ignore the mip chains) we can compute the texture footprint of the bundle at various points in the data pipeline:

*: Assumes transcoding to equivalent BCn formats at decompression time.

See the Settings and Quality Guide to learn more about various NTC settings and how they affect compression ratios and image quality.

Cooperative Vector and Inference

Decompressing texels with NTC requires reading the latent data corresponding to a given texture coordinate and then performing an inference operation by running it through a small Multi-Layer Perceptron (MLP) network whose weights are determined during compression and stored as part of the compressed bundle. While this operation is modest relative to the massive networks employed by many other deep learning applications it still carries a significant computational cost relative to the average pixel shader commonly seen in 3D rendering applications. Fortunately, NTC is able to benefit from new Cooperative Vector extensions for Vulkan and Direct3D 12 which allow pixel shaders to leverage the same hardware acceleration used in large network inference. On Ada- and Blackwell-class GPUs this provides a 2-4x improvement in inference throughput over competing optimal implementations that do not utilize these new extensions.

In order to provide robust backwards compatibility, fallback implementations of the inference code using the DP4a instructions or regular integer math have also been provided. This will allow for the decompression code to be executed reliably on any platform that supports at least Direct3D 12 Shader Model 6; however, there will be substantial performance improvements on newer GPUs. See System Requirements for more details.

WARNING: Pre-Release Feature Dependency for Direct3D 12

NTC texture decompression for DX12 applications, both on-load and on-sample, relies on a preview version of the Microsoft DirectX 12 Agility SDK, specifically, 1.721.0-preview. In order for the LinAlg (formerly Cooperative Vector) extensions to work, the application must enable the D3D12ExperimentalShaderModels feature, which requires that Windows is configured to be in the Developer Mode.

A pre-release NVIDIA GPU driver version 620.12 or later is required for Shader Model 6.10 (LinAlg) functionality.

All non-CoopVec versions of DX12 decompression, as well as all Vulkan versions including CoopVec, are OK to use for shipping.

The DX12 LinAlg support is for testing purposes only. DO NOT SHIP ANY PRODUCTS USING IT.

Choosing the inference interface

The SDK exposes hardware-accelerated inference generically as "Cooperative Vector," but the concrete interface it compiles and runs depends on the graphics API selected at run time (--dx12 or --vk):

On Direct3D 12 the Cooperative Vector shaders are compiled exclusively against the LinAlg (SM 6.10) interface; there is no separate legacy Cooperative Vector path for DX12. If the LinAlg preview is unavailable - for example, Windows is not in Developer Mode, or the driver is not a preview build - run with --no-coopVec to use the DP4a fallback, which works on any Shader Model 6 GPU. The older VK_NV_cooperative_vector interface remains available through the Vulkan backend (--vk).

Quick Start Guide

See the Build Guide for instructions on compiling the SDK.

To experiment with how Neural Texture Compression performs with different latent shapes and texture bundles, follow the NTC Explorer Guide. A few sample material sets have been provided in this package under the ./assets/materials folder.

To see how NTC works on a sample 3D scene follow the instructions in the NTC Renderer Guide for pre-processing a GLTF scene. A sample scene has been provided under the ./assets/models folder.

SDK Contents

System Requirements

Operating System: - Windows 10/11 x64 - Windows 11 ARM64 - Linux x64

Graphics APIs: - DirectX 12 - with preview Agility SDK for LinAlg support - Vulkan 1.3

GPU for NTC decompression on load and transcoding to BCn: - Minimum: Anything compatible with Shader Model 6 [*] - Recommended: NVIDIA Turing (RTX 2000 series) and newer.

GPU for NTC inference on sample: - Minimum: Anything compatible with Shader Model 6 (will be functional but very slow) [*] - Recommended: NVIDIA Ada (RTX 4000 series) and newer.

GPU for NTC compression: - Minimum: NVIDIA Turing (RTX 2000 series). - Recommended: NVIDIA Ada (RTX 4000 series) and newer.

[*] The oldest GPUs that the NTC SDK functionality has been validated on are NVIDIA GTX 1000 series, AMD Radeon RX 6000 series, Intel Arc A series.

For Cooperative Vector support on NVIDIA GPUs, please use the NVIDIA Graphics Driver preview version 620.12 or newer for DX12 (Shader Model 6.10 / LinAlg), or at least version 570 for Vulkan. On Windows ARM64 the required driver comes with the ARM64 CUDA Toolkit (see the Windows ARM64 section below). The preview driver can be downloaded using the following link (requires an NVIDIA Developer Program account):

• GeForce GPUs: https://developer.nvidia.com/downloads/assets/secure/geforce-drivers/620.12_gameready_win11_win10-dch_64bit_international.exe

For a list of software components needed to build the SDK, please refer to the Build Guide.

Known Issues

• GDeflate decompression with VK_NV_memory_decompression with the NVIDIA 590.26 Developer Preview driver crashes the GPU unless Nsight Aftermath is enabled.
• Running NTC SDK apps in D3D12 mode on non-Developer Preview NVIDIA drivers may result in messages like ERROR: D3D12CreateDevice failed, error code = 0x80070057. To work around that, add --no-coopVec.
• DX12 LinAlg inference is incompatible with the current version of AMD developer preview drivers (26.10.07.02).
• Inference on Feedback mode in the Rendering sample is broken on AMD GPUs.

Build Guide

NTC SDK supports Windows x64, Windows ARM64, and Linux x64 targets.

Windows x64

Building the NTC SDK on Windows requires the following components:

• Visual Studio 2022 (at least the build tools)
• Windows SDK (tested with 10.0.26100.0)
• CMake (tested with v3.31)
• CUDA SDK (tested with v13.3)

Follow the usual way of building CMake projects on Windows:

• Clone the project recursively: sh git clone --recursive https://github.com/NVIDIA-RTX/RTXNTC.git
• Building using the "x64 Native Tools Command Prompt for VS 2022": sh cd RTXNTC mkdir build cd build cmake .. cmake --build .
• Building using CMake GUI:
• Set "Where is the source code" to the RTXNTC folder
• Set "Where to build the binaries" to RTXNTC/build
• Configure a solution using Visual Studio 2022 tools for the x64 platform.
• Generate and open the solution in Visual Studio.
• Build.

Visual Studio Code with CMake Tools extension and Ninja build system works fine, too.

Windows ARM64

Windows ARM64 (for example, the NVIDIA RTX Spark) is supported as well. The requirements are the same as for Windows x64, with the following differences:

• The Visual Studio 2022 installation must include the ARM64 C++ build tools.
• A CUDA Toolkit build for Windo