Simple gguf LLM Qwen3-VL, Qwen3.5, Qwen3.6, Gemma4 and others model loader for Comfy-UI.

Why need this version?

This version was created to meet my requirements: 1. The model must support gguf (gguf models run faster than transformer models). 2. The model must support the Qwen3-VL, Qwen3.5, Qwen3.6, Gemma4 multimodal model. 3. The node should be easily adaptable to work with any new released model. 4. After running, the node must be completely cleared from memory, leaving no garbage behind. This is important. Next come very resource-intensive processes that require ALL the memory. (Yes, the model will have to be reloaded every time, but this is better than storing the model as dead weight while heavier tasks suffer from lack of memory and run slower). In the latest update added a new keep_vram mode, which allows you to keep the model from being unloaded from memory. Convenient for small models and batch modes. 5. No auto-loaded models. You can use any models you already have (from LM Studio etc). Just simply specify their path on the disk to config.

Last update:

Nightly (tests) - Added Bernini presets. - Add _user_prompt_template functionality. Now you can modify the user_prompt using a specified template (the name of which matches the system_preset) - Add variables input. You can now set any user placeholders in {} in the system and user prompts. - Added autocomplete placeholders width, height, image_num, ref_num, audio_num, frame_num, user_prompt. By default, placeholder replase is disabled for backward compatibility. It can be enabled by passing user variables to the variables input (just like the config input) or by using _user_prompt_template, or by forcing it by entering "enable_variables": true, in config. - Added the add_image_id, add_audio_id, and add_frame_id configurations, which allow you to number the corresponding content according to a specified template before inserting it. - An additional configuration file has been added to the following path: ComfyUI\user\SimpleQwenVL_configs\system_prompts_user.json. - improvement of video input (part 1) - fix UnicodeDecodeError error in subprocess - add node Ideogram 4 JSON Preview and Ideogram 4 JSON Swap XY Coordinates - present_penalty/presence_penalty issue

16.06.2026 - V3.9 - Fix f-string: unmatched caused by nested double quotes - Fix disappearance of "\n" line breaks in raw_mode

22.05.2026 - V3.8 - Added example qwen_vl_test_translate - Added modes: save1, save2, save3 - Added example qwen_vl_test_image_storytaler - Added utils: Simple Text To Batch, Simple Text Insert, Simple Text Replace, Simple Join Strings - Added simple LLM configurator - Improved error output

03.05.2026 - V3.7 - Added force_mmproj settings. - Added support for n_cpu_moe, cpu_moe. Requires llama_cpp_python update to 0.3.37+. See the limitations in the Speed ​​test and memory overflow problem section below. - Standard parameter names are now supported - Added debug calculate token/sec - Added options for running encoder (to obtain embeddings or conditioning) - Added video input (while llama.cpp doesn't have native support yet, you can pass a reduced set of frames, see example) - Added audio input (see example) - Added split_mode settings for multi GPU

04.04.2026 - V3.6 - Add Gemma4 support. - Fix raw_mode in text mode.

08.03.2026 - V3.5 - TurboQuants feature (for now requires a fork of llama.cpp) - Adding a new mode "raw_mode": true which allows you to set custom prompt templates. The Joycaption model now works correctly (see new configs below). - Three execution modes have been added: subprocess — inference runs in a separate process (safe, isolated); direct_clean — in the main process with model unloading after each run; keep_vram — the model remains in VRAM for repeated use. - Added config_override - the ability to add/override any configuration parameters via a text input directly in the node - Integrated json_repair to automatically repair invalid JSON in config_override and system_prompts_user.json - Expanded documentation on configuration fields and operating modes

04.03.2026 - V3.2 - Added support for Qwen3.5

Correct installation of llama-cpp-python:

Qwen3 support hasn't been added to the standard library, llama-cpp-python, which is downloaded via pip install llama-cpp-python - this didn't work. The standard version llama-cpp-python hasn't been updated for a long time. llama-cpp-python 0.3.16 last commit on Aug 15, 2025 and it doesn't support qwen3.

Check the version number of llama-cpp-python from JamePeng you're using: - Version 0.3.17 or latest supports qwen3-VL. - Version 0.3.30 or latest supports qwen3.5. - Version 0.3.35 or latest supports gemma4.

Variant 1 - Download WHL

Download WHL packages for your configuration

• https://github.com/JamePeng/llama-cpp-python/releases

For example:

cd *path_to_comfyui*\python_embeded

python -m pip install json_repair,colorama

python -m pip install temp\llama_cpp_python-0.3.18-cp313-cp313-win_amd64.whl

💡 WARNING: These ready-made basic VHLs may not have CPU acceleration implementations. Therefore, installing them may not yield any benefit from n_cpu_moe or cpu_moe. Use VHL with optimizations enabled, or better yet, compile the project yourself for your hardware.

💡 Tip: In subprocess mode, you can launch it immediately. In other modes, you need to restart Comfy-UI.

Variant 2 - Build from source code (I recommend this variant)

Installing software before compilation

1. Check that you have CUDA Toolkit installed. For example: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0
2. Try installing: https://developer.nvidia.com/cuda-downloads
3. Check that the PATH in Environment Variables includes the CUDA Toolkit bin folder (For example: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.0\bin).

4. After installing CUDA Toolkit, restart your computer.

5. Check that the NVIDIA Driver and CUDA Toolkit versions match (the driver can and most often should be newer than the CUDA Toolkit version): Run command in CMD nvidia-smi.

6. Check that you have Visual C++ Redistributable installed.

7. Try installing: https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170

8. Install both versions (x86 and x64).

9. Check that you have Visual Studio 2022 installed.

10. Install Visual Studio 2022.
11. Install the following packages (they will not be installed by default):

☑ Desktop development with C++ (in Workloads tab).

☑ MSVC v143 - VS 2022 C++ x64/x86 build tools (in Individual components tab).

☑ Windows 10/11 SDK (in Individual components tab).

☑ CMake tools for Visual Studio (in Individual components tab).

• The environment variable for MSVC is not added to the PATH by default. Run this command every time in your terminal before compiling: call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"

• If you use python_embeded for Comfy-UI, may need to add missing libs folders: python_embeded\include, python_embeded\libs (Not Lib\site-packages), python_embeded\DLLs:

• From here https://github.com/astral-sh/python-build-standalone/releases download Python appropriate version (for example cpython-3.13.11+20251217-x86_64-pc-windows-msvc-install_only.tar.gz)
• unzip and copy the necessary folders to python_embeded.

Build llama-cpp-python from source code

1. Clone the repositories using Git:
2. https://github.com/JamePeng/llama-cpp-python
3. https://github.com/ggml-org/llama.cpp

git clone https://github.com/JamePeng/llama-cpp-python.git
git clone https://github.com/ggml-org/llama.cpp.git

1. Move the second project llama.cpp\ in the llama-cpp-python\vendor\ folder

2. Automatically set the paths to MSVC (Windows only):

call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"

1. Optional: For fast build with Ninja

Using Ninja enables parallel compilation across CPU cores, significantly reducing build time (but may increase CPU temperature). Verify Ninja is installed with Visual Studio 2022:

ninja --version
1.12.1

• Configure environment variables (replace 32 with your desired number of cores):

set CMAKE_GENERATOR=Ninja
set MAX_JOBS=16

1. Go to the llama-cpp-python folder

cd *path_to_src*\llama-cpp-python

1. Set CUDA support and install the package:

*path_to_comfyui*\python -m pip install json_repair,colorama

set CMAKE_ARGS=-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release
*path_to_comfyui*\python_embeded\python -m pip install . 

✅ The command above is for embedded Python (typical for ComfyUI). Adjust the Python path if you're using a system or virtual environment.

Replace 120 with your сompute сapability number:

RTX 50-series (Blackwell) → 120
RTX 40-series → 89
RTX 30-series → 86
RTX 20-series → 75

⏱️ Build time: Without Ninja, compilation may take 30–60 minutes depending on your hardware.

⏱️ Build time: With Ninja, compilation may take 1–2 minutes depending on your hardware.

💡 Tip: In subprocess mode, you can launch it immediately. In other modes, you need to restart Comfu-ui.

Simple bat file for fast rebuild

cd llama-cpp-python\vendor\llama.cpp\
git pull
cd ..\..\
git pull --rebase
call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvars64.bat"
set CMAKE_GENERATOR=Ninja
set MAX_JOBS=16
set CMAKE_ARGS=-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=120 -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_BUILD_TYPE=Release
H:\ComfyUI128\python_embeded\python.exe -m pip install . --no-cache-dir --no-build-isolation
pause

✅ The command above is for embedded Python (typical for ComfyUI). Adjust the Python path if you're using a system or virtual environment.

Replace 120 with your сompute сapability number:

RTX 50-series (Blackwell) → 120
RTX 40-series → 89
RTX 30-series → 86
RTX 20-series → 75

💡 Tip: In subprocess mode, you can launch it immediately. In other modes, you need to restart Comfu-ui.

llama.cpp forks:

You can try installing various forks that implement new functionality that has not yet been added to the main library.

For example:

Launch a model with a huge context with TurboQuants

1. The standard library llama.cpp doesn't yet support TurboQuants, so for now we'll use this fork for llama.cpp (put fork in directory llama-cpp-python\vendor\llama.cpp): https://github.com/spiritbuun/llama-cpp-turboquant-cuda

2. Check branch: feature/turboquant-kv-cache

3. Then comes the standard compilation (There may be errors, as the project is completely new).

4. In Comfy-UI, you will need to enable this mode as follows (no new special models are required, the mechanism works with older models) options, by connecting a textbox to the config_override input with the following text:

"verbose": true,
"n_ctx": 262144,
"type_k": 41,
"type_v": 41

Where: 262144 - max context to model 41 - ggml type, the following options are available in the fork:

GGML_TYPE_TURBO3_0 = 41, // TurboQuant 3-bit KV cache: 2-bit PolarQuant + 1-bit QJL
GGML_TYPE_TURBO4_0 = 42, // TurboQuant 4-bit KV cache: 3-bit PolarQuant + 1-bit QJL
GGML_TYPE_TURBO2_0 = 43, // TurboQuant 2-bit KV cache: 2-bit PolarQuant, no QJL
in file llama-cpp-python\vendor\llama.cpp\ggml\include\ggml.h

1. Result verbose:

llama_kv_cache: size = 1792.00 MiB (262144 cells,   8 layers,  1/1 seqs), K (turbo3):  896.00 MiB, V (turbo3):  896.00 MiB

Context compression up to ~4x

1. Asymmetric mode (if you notice a drop in quality, because k-quant are more sensitive)

"type_k": 8, //Q8 (Not to be confused with quantization of model weights, this is quantization of attention)
"type_v": 41 //turbo3

CUDA Support

This project requires CUDA runtime libraries. They can be sourced from: - The CUDA Toolkit: https://developer.nvidia.com/cuda-downloads (recommended) - OR an existing PyTorch installation

💡 Tip: If you use ComfyUI, you likely already have PyTorch. In that case, you probably don't need to install the CUDA Toolkit separately — the necessary libraries will be found automatically.

💡 Tip: After installing CUDA Toolkit, restart your computer.

Installation of ComfyUI_Simple_Qwen3-VL-gguf:

1.Installation to custom_nodes - Use ComfyUI Manager and find ComfyUI_Simple_Qwen3-VL-gguf - OR copy this project to the folder path_to_comfyui\ComfyUI\custom_nodes

  cd path_to_comfyui\ComfyUI\custom_nodes
  git clone https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf

1. Restart ComfyUI. We check in the console that custom nodes are loading without errors.
2. Restarting the frontend (F5)

Implementation Features:

The node is split into two parts. All work is isolated in a subprocess. Why? To ensure everything is cleaned up and nothing unnecessary remains in memory after this node runs and llama.cpp. I've often encountered other nodes leaving something behind, and that's unacceptable to me.

💡 Update: The llama_python_cpp code has been improved and no longer leaks memory, so it is now possible to call llama_cpp directly.

Mode	Characteristics	Benefits
subprocess	Inference runs in a separate Python process. The model is loaded and u