Civit Image grabber 2.3

It downloads all the images and Videos from a provided Username, Model ID or Model TAG from CivitAI. Should the API not spit out all the data for all images then I'm sorry. The script can only download where data is provided.

The files are downloaded into a folder with the name of the user, ModelID or the TAG

Second Level is the Model Name with which the image was generated.

Videos without model metadata are placed in a dedicated videos/ subfolder.

Installation

1. Install Python 3 Ensure you have Python 3.8 or newer installed.

install Python3

1. Install Dependencies NEW requirements for users who already use the script

pip install -r requirements.txt

1. Migrate Existing Data
2. If you are using the previous version with a downloaded_images.json file and want to continue using your data for tracking, you must use the my tool migrate_json_to_sqlite.py to transfer your old data to the new database.
3. Make sure both migrate_json_to_sqlite.py and your old downloaded_images.json are in the same directory.
4. Run the migration tool from your terminal:

python migrate_json_to_sqlite.py

• Follow the prompts. It will create tracking_database.sqlite and offer to rename your old JSON file.

---

Usage

Interactive Mode

Run the script without any command-line arguments:

python civit_image_downloader.py

the script will ask you to:

1. Enter timeout value (seconds) [default: 60]:
2. Choose image quality (1=SD, 2=HD) [default: 1]:
3. Allow re-downloading tracked items? (1=Yes, 2=No) [default: 2]:
4. Skip video files? (y/n) [default: n]:
5. Choose mode (1=user, 2=model ID, 3=tag search, 4=model version ID):
6. Enter max concurrent downloads [default: 5]:
7. (Mode-specific prompts):
   • Mode 1: Enter username(s) (, separated):
   • Mode 1: Enter filter tag(s) (comma-separated, optional, press Enter to skip): (Optional: filter images by tags)
   • Mode 1: Disable prompt check? (y/n) [default: n]: (If filter tags are provided)
   • Mode 2: Enter model ID(s) (numeric, , separated):
   • Mode 3: Enter tags (, separated):
   • Mode 3: Disable prompt check? (y/n) [default: n]: (Check if tag words must be in the image prompt)
   • Mode 4: Enter model version ID(s) (numeric, , separated):
   • Mode 4: Enter filter tag(s) (comma-separated, optional, press Enter to skip): (Optional: filter by tags)
   • Mode 4: Disable prompt check? (y/n) [default: n]: (If filter tags are provided)

If you just hit enter it will use the Default values of that Option if it has a default value.

Command-Line Mode

Provide arguments directly on the command line. Unspecified arguments will use their defaults. --mode is required.

Available Arguments

• --timeout INT (Default: 60)
• --quality {1,2} (1=SD, 2=HD, Default: SD)
• --redownload {1,2} (1=Yes, 2=No, Default: 2)
• --mode {1,2,3,4} (Required)
• --tags TAGS (Comma-separated, required for Mode 3)
• --disable_prompt_check {y,n} (Default: n, works with Mode 1 (with filter_tags), Mode 3 and Mode 4)
• --username USERNAMES (Comma-separated, required for Mode 1)
• --model_id IDS (Comma-separated, numeric, required for Mode 2)
• --model_version_id IDS (Comma-separated, numeric, required for Mode 4)
• --filter_tags TAGS (Comma-separated, optional for Mode 1 and Mode 4, filters images by tags)
• --output_dir PATH (Default: "image_downloads")
• --semaphore_limit INT (Default: 5)
• --no_sort (Disables model subfolder sorting, Default: False/Sorting enabled)
• --no_videos (Skip video files, download images only, Default: False)
• --videos_only (Skip images, download video files only. Default: False)
• --max_path INT (Default: 240)
• --retries INT (Default: 2)
• --max_images INT (Limit total images downloaded, Default: unlimited)
• --max_per_model INT (Limit images per model in tag searches, Default: unlimited)
• --deep_scan (Enable deep scan for users with 50K+ images, Mode 1 only, Default: False)
• --debug (opt-in verbose logging. Default runs stay at INFO level to keep log files smaller.)

Examples

• Download HD images for user "artist1", allowing redownloads, higher concurrency: bash python civit_image_downloader.py --mode 1 --username "artist1" --quality 2 --redownload 1 --semaphore_limit 10
• Download SD images for models 123 and 456, using defaults for other options: bash python civit_image_downloader.py --mode 2 --model_id "123, 456"
• Download SD images for tag "sci-fi", disabling prompt check, no redownloads: bash python civit_image_downloader.py --mode 3 --tags "sci-fi" --disable_prompt_check y --redownload 2
• Download images from model version 123456, filtered by tags "anime" and "portrait": bash python civit_image_downloader.py --mode 4 --model_version_id "123456" --filter_tags "anime,portrait" --disable_prompt_check y
• Download only "anime" images from a specific username (Mode 1 + tag filter): bash python civit_image_downloader.py --mode 1 --username "artist123" --filter_tags "anime" --max_images 50
• Download only 100 images from user "artist1" (useful for testing or sampling): bash python civit_image_downloader.py --mode 1 --username "artist1" --max_images 100
• Download images from tag search with per-model limit (balanced sampling): bash python civit_image_downloader.py --mode 3 --tags "landscape" --max_per_model 50 --max_images 500
• To skip all video files: bash python civit_image_downloader.py --mode 1 --username "artist1" --no_videos
• Download ALL images from a user (50K+ images) using deep scan: bashhttps://civitai.red/articles/28369/two-front-doors-civitaicom-civitaired-and-whats-next python civit_image_downloader.py --mode 1 --username "Artist1" --deep_scan

Deep Scan

CivitAI's API caps pagination at ~50,000 images per user. Users with more than 50K images will silently get incomplete downloads. Pass --deep_scan to run additional passes that retrieve images beyond this limit using bi-directional pagination and per-model-version queries. Only applies to Mode 1 (username search). Without this flag, the script warns when a user hits the cap. For more technical information https://github.com/Confuzu/CivitAI_Image_grabber/tree/main#21-deep-scan----50k-api-pagination-cap-bypass

Mixed Mode

If only some arguments are provided (e.g., only --mode), the script will use the provided options and prompt the user for any missing inputs.

---

Folder Structure

The downloaded files will be organized within the specified --output_dir (default: image_downloads). Sorting (--no_sort flag) affects the structure inside the identifier folder.

With Sorting Enabled (Default)

image_downloads/
├── Username_Search/
│   └── [Username]/
│       ├── [Model Name Subfolder]/  # Based on image metadata 'Model' field
│       │   ├── [ImageID].jpeg       # or .png, .webp
│       │   └── [ImageID]_meta.txt
│       ├── videos/                  # Videos without parsable model metadata (.mp4, .webm)
│       │   ├── [ImageID].mp4        # or .webm
│       │   └── [ImageID]_no_meta.txt (or _meta.txt)
│       ├── invalid_metadata/        # For images with meta but no parsable 'Model' field
│       │   ├── [ImageID].jpeg
│       │   └── [ImageID]_meta.txt
│       └── no_metadata/             # For images (non-video) with no metadata found
│           ├── [ImageID].jpeg
│           └── [ImageID]_no_meta.txt
├── Model_ID_Search/
│   └── model_[ModelID]/
│       ├── [Model Name Subfolder]/
│       │   ├── [ImageID].jpeg
│       │   └── [ImageID]_meta.txt
│       ├── videos/                  # Videos without parsable model metadata (.mp4, .webm)
│       │   ├── [ImageID].mp4
│       │   └── [ImageID]_no_meta.txt (or _meta.txt)
│       ├── invalid_metadata/
│       │   ├── [ImageID].jpeg
│       │   └── [ImageID]_meta.txt
│       └── no_metadata/
│           ├── [ImageID].jpeg
│           └── [ImageID]_no_meta.txt
│
├── Model_Version_ID_Search/
│   └── modelVersion_[VersionID]/
│       ├── [Model Name Subfolder]/
│       │   ├── [ImageID].jpeg
│       │   └── [ImageID]_meta.txt
│       ├── videos/                  # Videos without parsable model metadata (.mp4, .webm)
│       │   ├── [ImageID].mp4
│       │   └── [ImageID]_no_meta.txt (or _meta.txt)
│       ├── invalid_metadata/
│       │   ├── [ImageID].jpeg
│       │   └── [ImageID]_meta.txt
│       └── no_metadata/
│           ├── [ImageID].jpeg
│           └── [ImageID]_no_meta.txt
└── Model_Tag_Search/
    └── [Sanitized_Tag_Name]/         # e.g., sci_fi_vehicle
        ├── model_[ModelID]/          # Folder for each model found under the tag
        │   ├── [Model Name Subfolder]/
        │   │   ├── [ImageID].jpeg
        │   │   └── [ImageID]_meta.txt
        │   ├── videos/               # Videos without parsable model metadata (.mp4, .webm)
        │   │   ├── [ImageID].mp4
        │   │   └── [ImageID]_no_meta.txt (or _meta.txt)
        │   ├── invalid_metadata/
        │   │   ├── [ImageID].jpeg
        │   │   └── [ImageID]_meta.txt
        │   └── no_metadata/
        │       ├── [ImageID].jpeg
        │       └── [ImageID]_no_meta.txt
        └── summary_[Sanitized_Tag_Name]_[YYYYMMDD].csv

With Sorting Disabled (--no_sort)

All images, videos and metadata files for a given identifier (username, model ID, model version ID, or model ID within a tag) are placed directly within that identifier's folder, without the [Model Name Subfolder], videos/, invalid_metadata, or no_metadata subdirectories.

image_downloads/
├── Username_Search/
│   └── [Username]/
│       ├── [ImageID].jpeg        # or .png, .webp
│       ├── [ImageID].mp4         # or .webm (videos)
│       ├── [ImageID]_meta.txt
│       └── [ImageID]_no_meta.txt
├── Model_ID_Search/
│   └── model_[ModelID]/
│       ├── [ImageID].jpeg
│       ├── [ImageID].mp4
│       └── ...
├── Model_Version_ID_Search/
│   └── modelVersion_[VersionID]/
│       ├── [ImageID].jpeg
│       ├── [ImageID].mp4
│       └── ...
└── Model_Tag_Search/
    └── [Sanitized_Tag_Name]/
        ├── model_[ModelID]/
        │   ├── [ImageID].jpeg
        │   ├── [ImageID].mp4
        │   ├── [ImageID]_meta.txt
        │   └── [ImageID]_no_meta.txt
        └── summary_[Sanitized_Tag_Name]_[YYYYMMDD].csv

---

Tracking Database (tracking_database.sqlite)

This file replaces the old JSON file. It stores a record of each downloaded image/video, including its path, quality, download date, associated tags (from Mode 3), original URL, and extracted checkpoint name (from metadata). You can explore this file using tools like "DB Browser for SQLite".

Migration Tool (migrate_json_to_sqlite.py)

If you are updating from a version using downloaded_images.json, run this separate Python script once in the same directory as your JSON file before using the main downloader. It will read the JSON and populate the new tracking_database.sqlite file.

python migrate_json_to_sqlite.py

---

Update History

2.3 API Retry Hardening, Deep Scan Recovery and videos_only

This update focuses on downloads where CivitAI's image API may return temporary DNS/connect/read failures or repeated 500/502/503/504 responses on cursor pages. A single failed cursor page can stop normal pagination early, so API page fetches now use stronger retry handling than regular image downloads. If a username download still hits a broken later cursor page after all retries, --deep_scan can now run recovery passes instead of stopping with only the partial normal pagination result. Persistent API-side 500 cursor failures can still leave gaps, but the script now retries more aggressively and can recover additional images through alternate scan paths. - API page fetches now use at least 10 attempts, even when the normal --retries value is lower. - DNS, connect, read, timeout, 429, and 50x API failures are retried before pagination is stopped. - For users with unstable or very large galleries, use --deep_scan.

Addresses GitHub Issue: #69

Added --debug for opt-in verbose logging. Default runs stay at INFO level to keep log files smaller. - For bug reports, add --debug and include the generated log file.

videos_only (Skip images, download video files only. Default: False)

2.2 Update

Civitai.com / Civitai.red split: - The API is still queried through CivitAI API hosts and now accepts pagination links from both civitai.com and civitai.red.

2.1 Deep Scan -- 50K API Pagination Cap Bypass

New --deep_scan flag for Mode 1 (username search) that retrieves images beyond the CivitAI API's 50K pagination cap.

The Problem

CivitAI's /api/v1/images endpoint uses cursor-based pagination that caps at ~50,000 items (250 pages x 200 items per page). After that, nextPage becomes null and pagination silently stops. For users with more than 50K images, this means a standard download only retrieves a fraction of their gallery. A user with 148,208 images would only get ~50,000 without deep scan.

How Deep Scan Works

Deep scan uses a 5-pass strategy to retrieve images beyond the 50K cap:

Pass	Strategy	Purpose
1	sort=Newest, nsfw=X	Initial pass (NSFW images, newest first)
2	sort=Oldest, nsfw=X	NSFW images from the other end
3	sort=Newest, no nsfw param	SFW images, newest first
4	sort=Oldest, no nsfw param	SFW images from the other end
5	Per-modelVersionId queries	Targeted gap-filling for specific models

Each pass paginates until the API stops returning resul