Afero: The Universal Filesystem Abstraction for Go

Afero is a powerful and extensible filesystem abstraction system for Go. It provides a single, unified API for interacting with diverse filesystems—including the local disk, memory, archives, and network storage.

Afero acts as a drop-in replacement for the standard os package, enabling you to write modular code that is agnostic to the underlying storage, dramatically simplifies testing, and allows for sophisticated architectural patterns through filesystem composition.

Why Afero?

Afero elevates filesystem interaction beyond simple file reading and writing, offering solutions for testability, flexibility, and advanced architecture.

🔑 Key Features:

• Universal API: Write your code once. Run it against the local OS, in-memory storage, ZIP/TAR archives, or remote systems (SFTP, GCS).
• Ultimate Testability: Utilize MemMapFs, a fully concurrent-safe, read/write in-memory filesystem. Write fast, isolated, and reliable unit tests without touching the physical disk or worrying about cleanup.
• Powerful Composition: Afero's hidden superpower. Layer filesystems on top of each other to create sophisticated behaviors:
  • Sandboxing: Use CopyOnWriteFs to create temporary scratch spaces that isolate changes from the base filesystem.
  • Caching: Use CacheOnReadFs to automatically layer a fast cache (like memory) over a slow backend (like a network drive).
  • Security Jails: Use BasePathFs to restrict application access to a specific subdirectory (chroot).
• os Package Compatibility: Afero mirrors the functions in the standard os package, making adoption and refactoring seamless.
• io/fs Compatibility: Fully compatible with the Go standard library's io/fs interfaces.

Installation

go get github.com/spf13/afero

import "github.com/spf13/afero"

Quick Start: The Power of Abstraction

The core of Afero is the afero.Fs interface. By designing your functions to accept this interface rather than calling os.* functions directly, your code instantly becomes more flexible and testable.

1. Refactor Your Code

Change functions that rely on the os package to accept afero.Fs.

// Before: Coupled to the OS and difficult to test
// func ProcessConfiguration(path string) error {
//     data, err := os.ReadFile(path)
//     ...
// }

import "github.com/spf13/afero"

// After: Decoupled, flexible, and testable
func ProcessConfiguration(fs afero.Fs, path string) error {
    // Use Afero utility functions which mirror os/ioutil
    data, err := afero.ReadFile(fs, path)
    // ... process the data
    return err
}

2. Usage in Production

In your production environment, inject the OsFs backend, which wraps the standard operating system calls.

func main() {
    // Use the real OS filesystem
    AppFs := afero.NewOsFs()
    ProcessConfiguration(AppFs, "/etc/myapp.conf")
}

3. Usage in Testing

In your tests, inject MemMapFs. This provides a blazing-fast, isolated, in-memory filesystem that requires no disk I/O and no cleanup.

func TestProcessConfiguration(t *testing.T) {
    // Use the in-memory filesystem
    AppFs := afero.NewMemMapFs()

    // Pre-populate the memory filesystem for the test
    configPath := "/test/config.json"
    afero.WriteFile(AppFs, configPath, []byte(`{"feature": true}`), 0644)

    // Run the test entirely in memory
    err := ProcessConfiguration(AppFs, configPath)
    if err != nil {
        t.Fatal(err)
    }
}

Afero's Superpower: Composition

Afero's most unique feature is its ability to combine filesystems. This allows you to build complex behaviors out of simple components, keeping your application logic clean.

Example 1: Sandboxing with Copy-on-Write

Create a temporary environment where an application can "modify" system files without affecting the actual disk.

// 1. The base layer is the real OS, made read-only for safety.
baseFs := afero.NewReadOnlyFs(afero.NewOsFs())

// 2. The overlay layer is a temporary in-memory filesystem for changes.
overlayFs := afero.NewMemMapFs()

// 3. Combine them. Reads fall through to the base; writes only hit the overlay.
sandboxFs := afero.NewCopyOnWriteFs(baseFs, overlayFs)

// The application can now "modify" /etc/hosts, but the changes are isolated in memory.
afero.WriteFile(sandboxFs, "/etc/hosts", []byte("127.0.0.1 sandboxed-app"), 0644)

// The real /etc/hosts on disk is untouched.

Example 2: Caching a Slow Filesystem

Improve performance by layering a fast cache (like memory) over a slow backend (like a network drive or cloud storage).

import "time"

// Assume 'remoteFs' is a slow backend (e.g., SFTP or GCS)
var remoteFs afero.Fs 

// 'cacheFs' is a fast in-memory backend
cacheFs := afero.NewMemMapFs()

// Create the caching layer. Cache items for 5 minutes upon first read.
cachedFs := afero.NewCacheOnReadFs(remoteFs, cacheFs, 5*time.Minute)

// The first read is slow (fetches from remote, then caches)
data1, _ := afero.ReadFile(cachedFs, "data.json")

// The second read is instant (serves from memory cache)
data2, _ := afero.ReadFile(cachedFs, "data.json")

Example 3: Security Jails (chroot)

Restrict an application component's access to a specific subdirectory.

osFs := afero.NewOsFs()

// Create a filesystem rooted at /home/user/public
// The application cannot access anything above this directory.
jailedFs := afero.NewBasePathFs(osFs, "/home/user/public")

// To the application, this is reading "/"
// In reality, it's reading "/home/user/public/"
dirInfo, err := afero.ReadDir(jailedFs, "/")

// Attempts to access parent directories fail
_, err = jailedFs.Open("../secrets.txt") // Returns an error

Real-World Use Cases

Build Cloud-Agnostic Applications

Write applications that seamlessly work with different storage backends:

type DocumentProcessor struct {
    fs afero.Fs
}

func NewDocumentProcessor(fs afero.Fs) *DocumentProcessor {
    return &DocumentProcessor{fs: fs}
}

func (p *DocumentProcessor) Process(inputPath, outputPath string) error {
    // This code works whether fs is local disk, cloud storage, or memory
    content, err := afero.ReadFile(p.fs, inputPath)
    if err != nil {
        return err
    }

    processed := processContent(content)
    return afero.WriteFile(p.fs, outputPath, processed, 0644)
}

// Use with local filesystem
processor := NewDocumentProcessor(afero.NewOsFs())

// Use with Google Cloud Storage
processor := NewDocumentProcessor(gcsFS)

// Use with in-memory filesystem for testing
processor := NewDocumentProcessor(afero.NewMemMapFs())

Treating Archives as Filesystems

Read files directly from .zip or .tar archives without unpacking them to disk first.

import (
    "archive/zip"
    "github.com/spf13/afero/zipfs"
)

// Assume 'zipReader' is a *zip.Reader initialized from a file or memory
var zipReader *zip.Reader 

// Create a read-only ZipFs
archiveFS := zipfs.New(zipReader)

// Read a file from within the archive using the standard Afero API
content, err := afero.ReadFile(archiveFS, "/docs/readme.md")

Serving Any Filesystem over HTTP

Use HttpFs to expose any Afero filesystem—even one created dynamically in memory—through a standard Go web server.

import (
    "net/http"
    "github.com/spf13/afero"
)

func main() {
    memFS := afero.NewMemMapFs()
    afero.WriteFile(memFS, "index.html", []byte("<h1>Hello from Memory!</h1>"), 0644)

    // Wrap the memory filesystem to make it compatible with http.FileServer.
    httpFS := afero.NewHttpFs(memFS)

    http.Handle("/", http.FileServer(httpFS.Dir("/")))
    http.ListenAndServe(":8080", nil)
}

Testing Made Simple

One of Afero's greatest strengths is making filesystem-dependent code easily testable:

func SaveUserData(fs afero.Fs, userID string, data []byte) error {
    filename := fmt.Sprintf("users/%s.json", userID)
    return afero.WriteFile(fs, filename, data, 0644)
}

func TestSaveUserData(t *testing.T) {
    // Create a clean, fast, in-memory filesystem for testing
    testFS := afero.NewMemMapFs()

    userData := []byte(`{"name": "John", "email": "john@example.com"}`)
    err := SaveUserData(testFS, "123", userData)

    if err != nil {
        t.Fatalf("SaveUserData failed: %v", err)
    }

    // Verify the file was saved correctly
    saved, err := afero.ReadFile(testFS, "users/123.json")
    if err != nil {
        t.Fatalf("Failed to read saved file: %v", err)
    }

    if string(saved) != string(userData) {
        t.Errorf("Data mismatch: got %s, want %s", saved, userData)
    }
}

Benefits of testing with Afero: - ⚡ Fast - No disk I/O, tests run in memory - 🔄 Reliable - Each test starts with a clean slate - 🧹 No cleanup - Memory is automatically freed - 🔒 Safe - Can't accidentally modify real files - 🏃 Parallel - Tests can run concurrently without conflicts

Backend Reference

Afero vs. io/fs (Go 1.16+)

Go 1.16 introduced the io/fs package, which provides a standard abstraction for read-only filesystems.

Afero complements io/fs by focusing on different needs:

• Use io/fs when: You only need to read files and want to conform strictly to the standard library interfaces.
• Use Afero when:
  • Your application needs to create, write, modify, or delete files.
  • You need to test complex read/write interactions (e.g., renaming, concurrent writes).
  • You need advanced compositional features (Copy-on-Write, Caching, etc.).

Afero is fully compatible with io/fs. You can wrap any Afero filesystem to satisfy the fs.FS interface using afero.NewIOFS:

import "io/fs"

// Create an Afero filesystem (writable)
var myAferoFs afero.Fs = afero.NewMemMapFs()

// Convert it to a standard library fs.FS (read-only view)
var myIoFs fs.FS = afero.NewIOFS(myAferoFs)

Third-Party Backends & Ecosystem

The Afero community has developed numerous backends and tools that extend the library's capabilities. Below are curated, well-maintained options organized by maturity and reliability.

Featured Community