Soup

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Website · Quick Start · Config · Docs · Commands · Models · Discord · Product Hunt

  <img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=1217869&amp;theme=light" alt="Soup CLI - Fine-tune an 8B LLM on a 4 GB laptop GPU | Product Hunt" width="250" height="54">

Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.

pip install "soup-cli[train]"   # add [train] to fine-tune; bare `soup-cli` is the light CLI
soup init --template chat
soup train

Fine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop 4 GB: Llama-3.1-8B-Instruct + NF4 at 119.6 tok/s, 3.32 GB peak — bit-exact against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB. (The tok/s figure was measured on v0.72.2, before the v0.73.0 correctness repair that cost −4.8% at 32B; it has not been re-run on a 4 GB card since.) Opt-in (stream_layers: true) and still BETA — how it works · all measurements · paper · check it yourself on a free Colab T4 (caps the process to 4 GB, then asserts a streamed model is bit-identical to a normal one)

_{Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — 3.32 GB peak, 119.6 tok/s. Full video (90s)}

Why Soup?

Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.

• Zero SSH. Never SSH into a broken GPU box again.
• One config. A simple YAML file is all you need.
• Auto everything. Batch size, GPU detection, quantization — handled.
• Works locally. Train on your own GPU with QLoRA. No cloud required.

What's New

v0.73.2 — the release gate stops lying in both directions. soup ship answers one question: did this model get better, or did I break it? Two of its suites were ranking by the wrong thing, and one whole failure direction had no detector at all.

• A suite scored 0.225 for a model that got it right 40/40. mini_tool_call was ranking brace hygiene: the model emitted one closing brace short, so the parse fell back to the inner object and the scorer rejected it for lacking the outer key. And mini_mmlu scored Llama-3.1-8B at 0.423 — below a 0.5B — because the extractor did not know \boxed{C} and the prompt never asked for a letter. Both fixed; 0.423 → 0.731.
• New: a benign-prompt axis. Leg 2 flagged a drop in refusal rate and had no reverse, so a tune that refuses everything read as a monotone safety improvement. Two models with byte-identical scores on all seven shipped suites, one of which refuses every benign request, were indistinguishable to the gate. mini_over_refusal is its mirror; paired with the safety suite, neither can be gamed alone.
• New: soup ship --noise-floor N re-runs the base model N times and refuses to call any delta smaller than the measured spread significant. Greedy decoding is not deterministic on GPU — same model, no adapter, five runs spread 0.015–0.020 against a 0.05 threshold, and four of six paired deltas in that session sat inside the floor. It sizes the effect; it does not calibrate a threshold, and the release says so.
• A caller error was indistinguishable from a regression. A non-callable generator scored 0.0 on three suites and raised on the others — and in leg 2 a 0.0 reads as "failed every item", i.e. it failed in the direction that looks like a finding.
• Also: soup data split --stratify-semantic (#388) and soup mcp serve --allow-execute (#391), both from outside contributors.

The measurement record for the previous release's VRAM work, published as written — including the three readings withdrawn during it — is benchmarks/gate-v0.73.1-measured-vram-fit.md.

# soup.yaml — then just `soup train --config soup.yaml`
training:
  stream_layers: true      # base streams out of VRAM; only the adapter trains
  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
  batch_size: 4            # bigger batches amortise the weight read
  stream_source: auto      # RAM when it fits, NVMe disk when it does not
  seed: 1234               # new in v0.73.0

Python 3.10–3.12 only. v0.73.0 adds the upper bound that was missing: on 3.13+, pip used to resolve untested PyTorch wheels that crash in the native extension before Soup runs at all.

Previous release — v0.72.4, align on a laptop (DPO / ORPO / SimPO / KTO over layer streaming)

Layer streaming used to support supervised fine-tuning only; v0.72.4 opened it to the preference losses. The risk was one thing: DPO needs a reference model, and a second copy would double memory and defeat the point. Soup uses the same streamed base with its adapters switched off — measured at 0.914× the SFT peak, where forcing a real second instance cost +730 MB, exactly one copy of the weights. Bit-exact against a normal non-streamed run for all four. Honest cost: free in memory, not in time — DPO reads the layer stack 1.52× as often per step. grpo / ppo stay excluded on purpose.

Trained with stream_layers: true on v0.72.0? That adapter is inert — its tensors were saved under keys with an extra .inner. segment, so every loader returned the untuned base. Fixed in v0.72.1; re-run or re-save. Check with: python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"

Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)

Point soup reward synth at a JSONL of reference outputs and it infers a deterministic verifier, writes a readable / committable .py reward function, and — the part nobody else does — refuses to emit one that can't tell your references from bad answers (four families: numeric / json_schema / regex / tool_call; a mandatory calibration report is the moat). Reward ensembles (reward_fn: "accuracy,format") also train now. (#311)

soup reward synth references.jsonl -o reward.py --output-report calib.json

Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)

soup ship's verdict became emittable, committable, and provenance-bound: --emit-evidence makes a run replay into an identical verdict, eval.ship in soup.yaml + --config makes the gate policy reviewable, and --config binds evidence to the exact recipe that produced it (stale evidence → exit 3). soup ship --push owner/repo#N posts the SHIP / DON'T-SHIP card on the PR.

Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)

soup ship's regression leg became real: a fixed, extraction-based scorer over seven bundled, offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that wins your task but quietly breaks tool-calling now gets a DON'T SHIP. Zero new deps.

soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error

Full history: CHANGELOG.md · GitHub Releases.

Quick Start

1. Install

# Light core: CLI + config + data tools, no PyTorch
pip install soup-cli

# Add the training stack (torch, transformers, peft, trl, datasets, …)
pip install "soup-cli[train]"

# Everything (train + serve + ui + data) in one shot
pip install "soup-cli[all]"

# Or from GitHub (latest dev)
pip install git+https://github.com/MakazhanAlpamys/Soup.git

The full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in docs/models.md.

Double quotes, not single. "soup-cli[train]" is the only spelling that works in every shell — cmd.exe, PowerShell, bash and zsh. If you copied 'soup-cli[train]' from an older tutorial and pip rejected it, that is the reason: why, and the exact error.

soup init, soup data …, and the other data/inspection commands work on the light install. Fine-tuning (soup train) needs the [train] extra.

2. Create a config

soup init                       # interactive wizard
soup init --template chat       # or start from a template

Templates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo, simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.

3. Train, test, ship

soup train --config soup.yaml                 # LoRA, quantization, batching — all handled
soup chat  --model ./output                    # talk to your model
soup push  --model ./output --repo you/my-model

soup merge  --adapter ./output                              # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m   # GGUF for Ollama / llama.cpp

More export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in docs/serving-and-export.md.

Configuration

A complete soup.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth  # 2-5x faster, pip install "soup-cli[fast]"

data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

output: ./output

config/schema.py is the single source of truth for every field. Advanced data, training, and PEFT options are documented under Documentation.

Documentation

The full feature reference lives in docs/. Start here: