github.com/containers/ramalama @v0.23.0

RamaLama strives to make working with AI simple, straightforward, and familiar by using OCI containers.

Description

RamaLama is an open-source tool that simplifies the local use and serving of AI models for inference from any source through the familiar approach of containers. It allows engineers to use container-centric development patterns and benefits to extend to AI use cases.

RamaLama eliminates the need to configure the host system by instead pulling a container image specific to the GPUs discovered on the host system, and allowing you to work with various models and platforms.

• Eliminates the complexity for users to configure the host system for AI.
• Detects and pulls an accelerated container image specific to the GPUs on the host system, handling dependencies and hardware optimization.
• RamaLama supports multiple AI model registries, including OCI Container Registries.
• Models are treated similarly to how Podman and Docker treat container images.
• Use common container commands to work with AI models.
• Run AI models securely in rootless containers, isolating the model from the underlying host.
• Keep data secure by defaulting to no network access and removing all temporary data on application exits.
• Interact with models via REST API or as a chatbot.

Install

Install on macOS (Self-Contained Installer)

Download the self-contained macOS installer that includes Python and all dependencies:

1. Download the latest .pkg installer from Releases
2. Double-click to install, or run: sudo installer -pkg RamaLama-*-macOS-Installer.pkg -target /

See macOS Installation Guide for detailed instructions.

Install on Fedora

RamaLama is available in Fedora and later. To install it, run:

sudo dnf install ramalama

Fedora Silverblue and Toolbox

On Fedora Silverblue and other immutable variants, the system is read-only. You can run RamaLama in either of these ways:

• Toolbox: Create a Toolbox container and install RamaLama inside it (pip install ramalama or dnf install ramalama if available). Use the same Podman (or Docker) from the host so that RamaLama can start model containers; ensure the container engine is installed and that the toolbox has access to the host's container engine (e.g., by bind-mounting the socket or by configuring the toolbox to use the host's podman command).
• Host Installation or Toolbox with Host Access: Install RamaLama on the host via rpm-ostree install ramalama if the package is available for your image, or run RamaLama from a toolbox, ensuring Podman/Docker is available and the model store is on a writable location (e.g., your home directory).

The model store defaults to ~/.local/share/ramalama, which is writable on Silverblue.

Install via PyPI

RamaLama is available via PyPI at https://pypi.org/project/ramalama

pip install ramalama

Install script (Linux and macOS)

Install RamaLama by running:

curl -fsSL https://ramalama.ai/install.sh | bash

Install on Windows

RamaLama supports Windows with Docker Desktop or Podman Desktop:

pip install ramalama

Requirements: - Python 3.9 or later - Docker Desktop or Podman Desktop with WSL2 backend - For GPU support, see NVIDIA GPU Setup for WSL2

Note: Windows support requires running containers via Docker/Podman. The model store uses hardlinks (no admin required) or falls back to file copies if hardlinks are unavailable.

Uninstall

Uninstall via pip

If you installed RamaLama using pip, you can uninstall it with:

pip uninstall ramalama

Uninstall on Fedora

If you installed RamaLama using DNF:

sudo dnf remove ramalama

Uninstall on macOS (Self-Contained Installer)

To remove RamaLama installed via the .pkg installer:

# Remove the executable
sudo rm /usr/local/bin/ramalama

# Remove configuration and data files
sudo rm -rf /usr/local/share/ramalama

# Remove man pages (optional)
sudo rm /usr/local/share/man/man1/ramalama*.1
sudo rm /usr/local/share/man/man5/ramalama*.5
sudo rm /usr/local/share/man/man7/ramalama*.7

# Remove shell completions (optional)
sudo rm /usr/local/share/bash-completion/completions/ramalama
sudo rm /usr/local/share/fish/vendor_completions.d/ramalama.fish
sudo rm /usr/local/share/zsh/site-functions/_ramalama

See the macOS Installation Guide for more details.

Remove User Data and Configuration

After uninstalling RamaLama using any method above, you may want to remove downloaded models and configuration files:

# Remove downloaded models and data (can be large)
rm -rf -- "${XDG_DATA_HOME:-~/.local/share}/ramalama"

# Remove configuration files
rm -rf -- "${XDG_CONFIG_HOME:-~/.config}/ramalama"

# If you ran RamaLama as root, also remove:
sudo rm -rf /var/lib/ramalama

Note: The model data directory (by default ~/.local/share/ramalama) can be quite large depending on how many models you've downloaded. Make sure you want to remove these files before running the commands above.

Accelerated images

Accelerator	Image
GGML_VK_VISIBLE_DEVICES (or CPU)	quay.io/ramalama/ramalama
HIP_VISIBLE_DEVICES	quay.io/ramalama/rocm
CUDA_VISIBLE_DEVICES	quay.io/ramalama/cuda
ASAHI_VISIBLE_DEVICES	quay.io/ramalama/asahi
INTEL_VISIBLE_DEVICES	quay.io/ramalama/intel-gpu
ASCEND_VISIBLE_DEVICES	quay.io/ramalama/cann
MUSA_VISIBLE_DEVICES	quay.io/ramalama/musa

GPU support inspection

On first run, RamaLama inspects your system for GPU support, falling back to CPU if none are present. RamaLama uses container engines like Podman or Docker to pull the appropriate OCI image with all necessary software to run an AI Model for your system setup.

How does RamaLama select the right image?

After initialization, RamaLama runs AI Models within a container based on the OCI image. RamaLama pulls container images specific to the GPUs discovered on your system. These images are tied to the minor version of RamaLama. - For example, RamaLama version 1.2.3 on an NVIDIA system pulls quay.io/ramalama/cuda:1.2. To override the default image, use the --image option.

RamaLama then pulls AI Models from model registries, starting a chatbot or REST API service from a simple single command. Models are treated similarly to how Podman and Docker treat container images.

Hardware Support

Hardware	Enabled
CPU	✓
Apple Silicon GPU (Linux / Asahi)	✓
Apple Silicon GPU (macOS)	✓ llama.cpp or MLX
Apple Silicon GPU (podman-machine)	✓
Nvidia GPU (cuda)	✓ See note below
AMD GPU (rocm, vulkan)	✓
Ascend NPU (Linux)	✓
Intel ARC GPUs (Linux)	✓ See note below
Intel GPUs (vulkan / Linux)	✓
Moore Threads GPU (musa / Linux)	✓ See note below
Windows (with Docker/Podman)	✓ Requires WSL2

Nvidia GPUs

On systems with NVIDIA GPUs, see ramalama-cuda documentation for the correct host system configuration.

Intel GPUs

The following Intel GPUs are auto-detected by RamaLama:

GPU ID	Description
0xe20b	Intel® Arc™ B580 Graphics
0xe20c	Intel® Arc™ B570 Graphics
0x7d51	Intel® Graphics - Arrow Lake-H
0x7dd5	Intel® Graphics - Meteor Lake
0x7d55	Intel® Arc™ Graphics - Meteor Lake

See the Intel hardware table for more information.

Moore Threads GPUs

On systems with Moore Threads GPUs, see ramalama-musa documentation for the correct host system configuration.

MLX Runtime (macOS only)

The MLX runtime provides optimized inference for Apple Silicon Macs. MLX requires: - macOS operating system - Apple Silicon hardware (M1, M2, M3, or later) - Usage with --nocontainer option (containers are not supported) - The mlx-lm uv package installed on the host system as a uv tool

To install and run Phi-4 on MLX, use uv. If uv is not installed, you can install it with curl -LsSf https://astral.sh/uv/install.sh | sh:

uv tool install mlx-lm
# or upgrade to the latest version:
uv tool upgrade mlx-lm

ramalama --runtime=mlx serve hf://mlx-community/Unsloth-Phi-4-4bit

Default Container Engine

When both Podman and Docker are installed, RamaLama defaults to Podman. The RAMALAMA_CONTAINER_ENGINE=docker environment variable can override this behaviour. When neither are installed, RamaLama will attempt to run the model with software on the local system.

Security

Test and run your models more securely

Because RamaLama defaults to running AI models inside rootless containers using Podman or Docker, these containers isolate the AI models from information on the underlying host. With RamaLama containers, the AI model is mounted as a volume into the container in read-only mode.

This results in the process running the model (llama.cpp or vLLM) being isolated from the host. Additionally, since ramalama run uses the --network=none option, the container cannot reach the network and leak any information out of the system. Finally, containers are run with the --rm option, which means any content written during container execution is deleted when the application exits.

Here’s how RamaLama delivers a robust security footprint:

• Container Isolation – AI models run within isolated containers, preventing direct access to the host system.
• Read-Only Volume Mounts – The AI model is mounted in read-only mode, which means that processes inside the container cannot modify the host files.
• No Network Access – ramalama run is executed with --network=none, meaning the model has no outbound connectivity for which information can be leaked.
• Auto-Cleanup – Containers run with --rm, wiping out any temporary data once the session ends.
• Drop All Linux Capabilities – No access to Linux capabilities to attack the underlying host.
• No New Privileges – Linux Kernel feature that disables container processes from gaining additional privileges.

Transports

RamaLama supports multiple AI model registries types called transports.

Supported transports

Transports	Web Site
HuggingFace	huggingface.co
ModelScope	modelscope.cn
Ollama	ollama.com
RamaLama Labs Container Registry	ramalama.com
OCI Container Registries	opencontainers.org
	Examples: quay.io, Docker Hub, Pulp, and Artifactory

Default Transport

Models specified in the Hugging Face <org>/<model> format are automatically pulled from Hugging Face. For models specified without an organization (e.g. granite-code), RamaLama currently defaults to the Ollama transport. Note: Ollama models are no longer compatible with llama.cpp, and support for the Ollama transport will be remo