github.com/ARahim3/mlx-tune @v0.6.0

Fine-tune LLMs, Vision, Audio, and OCR models on your Mac

SFT, DPO, GRPO, Vision, TTS, STT, Embedding, and OCR fine-tuning — natively on MLX. Unsloth-compatible API.

Documentation · Quick Start · Training Methods · Examples · Status

---

[!NOTE] Name Change: This project was originally called unsloth-mlx. Since it's not an official Unsloth project and to avoid any confusion, it has been renamed to mlx-tune. The vision remains the same — bringing the Unsloth experience to Mac users via MLX. If you were using unsloth-mlx, simply switch to pip install mlx-tune and update your imports from unsloth_mlx to mlx_tune.

[!NOTE] Why I Built This (A Personal Note)

I rely on Unsloth for my daily fine-tuning on cloud GPUs—it's the gold standard for me. But recently, I started working on a MacBook M4 and hit a friction point: I wanted to prototype locally on my Mac, then scale up to the cloud without rewriting my entire training script.

Since Unsloth relies on Triton (which Macs don't have, yet), I couldn't use it locally. I built mlx-tune to solve this specific "Context Switch" problem. It wraps Apple's native MLX framework in an Unsloth-compatible API.

The goal isn't to replace Unsloth or claim superior performance. The goal is code portability: allowing you to write FastLanguageModel code once on your Mac, test it, and then push that exact same script to a CUDA cluster. It solves a workflow problem, not just a hardware one.

This is an "unofficial" project built by a fan, for fans who happen to use Macs. It's helping me personally, and if it helps others like me, then I'll have my satisfaction.

Why MLX-Tune?

Bringing the Unsloth experience to Mac users via Apple's MLX framework.

• 🚀 Fine-tune LLMs, VLMs, TTS, STT & Embeddings locally on your Mac (M1/M2/M3/M4/M5)
• 💾 Leverage unified memory (up to 512GB on Mac Studio)
• 🔄 Unsloth-compatible API - your existing training scripts just work!
• 📦 Export anywhere - HuggingFace format, GGUF for Ollama/llama.cpp
• 🎙️ Audio fine-tuning - 5 TTS models (Orpheus, OuteTTS, Spark, Sesame, Qwen3-TTS) + 7 STT models (Whisper, Moonshine, Qwen3-ASR, NVIDIA Canary, Voxtral, Voxtral Realtime, NVIDIA Parakeet TDT)

# Unsloth (CUDA)                        # MLX-Tune (Apple Silicon)
from unsloth import FastLanguageModel   from mlx_tune import FastLanguageModel
from trl import SFTTrainer              from mlx_tune import SFTTrainer

# Rest of your code stays exactly the same!

What This Is (and Isn't)

This is NOT a replacement for Unsloth or an attempt to compete with it. Unsloth is incredible - it's the gold standard for efficient LLM fine-tuning on CUDA.

This IS a bridge for Mac users who want to: - 🧪 Prototype locally - Experiment with fine-tuning before committing to cloud GPU costs - 📚 Learn & iterate - Develop your training pipeline with fast local feedback loops - 🔄 Then scale up - Move to cloud NVIDIA GPUs + original Unsloth for production training

Local Mac (MLX-Tune)       →     Cloud GPU (Unsloth)
   Prototype & experiment          Full-scale training
   Small datasets                  Large datasets
   Quick iterations                Production runs

Project Status

🧠 v0.6.0 — JEPA comes to Apple Silicon. Yann LeCun's Joint-Embedding Predictive Architecture, the whole family: - LeJEPA — pretrain a Vision Transformer from scratch with the heuristics-free SIGReg objective (no EMA teacher, no predictor, no stop-gradient). - I-JEPA — load Meta's pretrained image encoder (facebook/ijepa_*) and fine-tune with frozen / LoRA / full classification. - V-JEPA 2 — load Meta's pretrained video world model (facebook/vjepa2-*): video classification, clip features, the masked-latent predictor (predict_latents + latent_energy — anticipate the rest of a clip, score how surprising the actual future was), and Meta's fine-tuned SSv2 action classifiers (174 classes, zero training). - LLM-JEPA — bring the JEPA objective to LLM fine-tuning (arXiv 2509.14252): augment next-token prediction with a JEPA term that aligns two "views" of the same item (e.g. a description and its code). First-on-MLX; the artifact is a normal LoRA model.

New entry points: FastJEPAModel, FastVideoJEPAModel, LLMJEPATrainer. See the JEPA docs.

Feature	Status	Notes
SFT Training	✅ Stable	Native MLX training
Model Loading	✅ Stable	Any HuggingFace model (quantized & non-quantized)
Save/Export	✅ Stable	HF format, GGUF (see limitations)
DPO Training	✅ Stable	Full DPO loss
ORPO Training	✅ Stable	Full ORPO loss
GRPO Training	✅ Stable	Multi-generation + reward
KTO Training	✅ Stable	Binary feedback + KTOConfig
SimPO Training	✅ Stable	No ref model + SimPOConfig
Chat Templates	✅ Stable	16 models (llama, gemma, qwen, phi, mistral)
Response-Only Training	✅ Stable	train_on_responses_only()
Multi-turn Merging	✅ Stable	to_sharegpt() + conversation_extension
Column Mapping	✅ Stable	apply_column_mapping() auto-rename
Dataset Config	✅ Stable	HFDatasetConfig structured loading
Vision Models	✅ Stable	Full VLM fine-tuning via mlx-vlm (Gemma 4, Qwen3.5, PaliGemma, LLaVA, Pixtral)
Gemma 4 Audio	✅ Stable	E2B/E4B STT/ASR via Conformer audio tower + optional audio LoRA
MoE Fine-Tuning	✅ Stable	Arcee Trinity-Nano (AFMoE), Gemma 4 26B-A4B, Qwen3.5-35B-A3B, Phi-3.5-MoE, Mixtral, DeepSeek, 39+ architectures
TTS Fine-Tuning	✅ Stable	Orpheus, OuteTTS, Spark-TTS, Sesame/CSM, Qwen3-TTS
STT Fine-Tuning	✅ Stable	Whisper, Moonshine, Qwen3-ASR, Canary, Voxtral, Voxtral Realtime (streaming), Parakeet TDT (CTC/RNN-T/TDT losses + auto vocab extension)
convert()	✅ Stable	HF → MLX conversion (LLM, TTS, STT)
Embedding Fine-Tuning	✅ Stable	BERT, ModernBERT, Qwen3-Embedding, Harrier (InfoNCE/contrastive)
OCR Fine-Tuning	✅ Stable	DeepSeek-OCR, GLM-OCR, olmOCR, Qwen-VL, Pixtral + CER/WER metrics
JEPA / LeJEPA	✅ Stable	From-scratch ViT SSL (LeJEPA: SIGReg, no EMA/predictor/stop-grad)
I-JEPA (pretrained)	✅ Stable	Meta facebook/ijepa_* → MLX (HF-parity verified); frozen / LoRA / full classification; linear / kNN / attentive probes; warm-start SSL
V-JEPA 2 (video)	✅ Stable	Meta facebook/vjepa2-* video world model → MLX (Conv3d + 3D RoPE, HF-parity verified); video classification + probes; predictor ported (predict_latents/latent_energy anticipation); pretrained SSv2 classifiers load directly
LLM-JEPA	✅ Stable	JEPA objective for LLM fine-tuning (arXiv 2509.14252): NTP + view-alignment (cosine/l2/mse/infonce), [PRED] slots; LLMJEPATrainer, artifact is a normal LoRA model
JEPA dense / regression	✅ Stable	for_image_regression (object counting) + for_dense_prediction (depth maps, segmentation) — I-JEPA's own headline tasks; frozen / LoRA / full
JEPA scale-up	✅ Stable	Non-224 pretrained I-JEPA (bicubic pos-embed interpolation) + ImageFolderDataset streaming + resumable checkpoints
LeWM (world model)	✅ Stable	LeWorldModel (arXiv 2603.19312): trainable latent world model from pixels (SIGReg, no stop-grad) + CEM/MPC planning; FastWorldModel, plan_cem
LFM2 Support	✅ Stable	Liquid AI LFM2/LFM2.5 (350M-24B, hybrid conv+GQA, Thinking)
Continual Pretraining	✅ Stable	CPTTrainer with decoupled LR, embed_tokens/lm_head, full-weight mode
push_to_hub()	✅ Stable	Upload to HuggingFace Hub
PyPI Package	✅ Available	uv pip install mlx-tune

Installation

# Using uv (recommended - faster and more reliable)
uv pip install mlx-tune

# With audio support (TTS/STT fine-tuning)
uv pip install 'mlx-tune[audio]'
brew install ffmpeg  # system dependency for audio codecs

# Or using pip
pip install mlx-tune

# From source (for development)
git clone https://github.com/ARahim3/mlx-tune.git
cd mlx-tune
uv pip install -e .

Quick Start

from mlx_tune import FastLanguageModel, SFTTrainer, SFTConfig
from datasets import load_dataset

# Load any HuggingFace model (1B model for quick start)
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="mlx-community/Llama-3.2-1B-Instruct-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=16,
)

# Load a dataset (or create your own)
dataset = load_dataset("yahma/alpaca-cleaned", split="train[:100]")

# Train with SFTTrainer (same API as TRL!)
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=SFTConfig(
        output_dir="outputs",
        per_device_train_batch_size=2,
        learning_rate=2e-4,
        max_steps=50,
    ),
)
trainer.train()

# Save (same API as Unsloth!)
model.save_pretrained("lora_model")  # Adapters only
model.save_pretrained_merged("merged", tokenizer)  # Full model (16-bit)
model.save_pretrained_gguf("model", tokenizer)  # GGUF (see note below)

[!NOTE] Merging into a 4-bit base: the default save_method="merged_16bit" dequantizes the base before fusing, so the fine-tune is preserved exactly. save_method="merged_4bit" keeps the base quantized and re-quantizes the fused weights — smaller on disk, but a weak LoRA delta (low LR / few steps) can be rounded away. Prefer the 16-bit default, or keep the adapter and use model.load_adapter(...) at inference time.

[!NOTE] GGUF Export: Works with non-quantized base models. If using a 4-bit model (like above), see Known Limitations for workarounds.

Chat Templates & Response-Only Training

from mlx_tune import get_chat_template, train_on_responses_only

# Apply chat template (supports llama-3, gemma, qwen, phi, mistral, etc.)
tokenizer = get_chat_template(tokenizer, chat_template="llama-3")

# Or auto-detect from model name
tokenizer = get_chat_template(tokenizer, chat_template="auto")

# Train only on responses (not prompts) - more efficient!
trainer = train_on_responses_only(
    trainer,
    instruction_part="<|start_header_id|>user<|end_header_id|>\n\n",
    response_part="<|start_header_id|>assistant<|end_header_id|>\n\n",
)

Vision Model Fine-Tuning (NEW!)

Fine-tune vision-language models like Gemma 4, Qwen3.5 on image+text tasks:

from mlx_tune import FastVisionModel, UnslothVisionDataCollator, VLMSFTTrainer
from mlx_tune.vlm import VLMSFTConfig

# Load a vision model
model, processor = FastVisionModel.from_pretrained(
    "mlx-community/Qwen3.5-0.8B-bf16",
)

# Add LoRA (same params as Unsloth!)
model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers=True,
    finetune_language_layers=True,
    r=16, lora_alpha=16,
)

# Train on image-text data
FastVisionModel.for_training(model)
trainer = VLMSFTTrainer(
    model=model,
    tokenizer=processor,
    data_collator=UnslothVisionDataCollator(model, processor),
    train_dataset=dataset,
    args=VLMSFTConfig(max_steps=30, learning_rate=2e-4),
)
trainer.train()

See examples/38_gemma4_vision_finetuning.py for Gemma 4 vision fine-tuning, examples/39_gemma4_text_to_sql.py for text-only fine-tuning through the VLM path, examples/10_qwen35_vision_finetuning.py for Qwen3.5, or examples/26_vision_grpo_training.py for Vision GRPO reasoning.

Gemma 4 Audio Fine-Tuning

Fine-tune Gemma 4 E2B/E4B for speech-to-text and audio understanding. The 12-layer Conformer audio tower processes 16kHz audio — no separate STT model needed:

```python from mlx_tune import FastVisionModel, UnslothVisionDataCollator, VLMSFTTrainer from mlx_tune.vlm import VLMSFTConfig

model, processor = FastVisionMode