imaginary

Fast HTTP microservice written in Go for high-level image processing backed by bimg and libvips. imaginary can be used as private or public HTTP service for massive image processing with first-class support for Docker & Fly.io. It's almost dependency-free and only uses net/http native package without additional abstractions for better performance.

Supports multiple image operations exposed as a simple HTTP API, with additional optional features such as API token authorization, URL signature protection, HTTP traffic throttle strategy and CORS support for web clients.

imaginary can read images from HTTP POST payloads, server local path or remote HTTP servers, supporting JPEG, PNG, WEBP, HEIF, and optionally TIFF, PDF, GIF and SVG formats if libvips@8.3+ is compiled with proper library bindings.

imaginary is able to output images as JPEG, PNG and WEBP formats, including transparent conversion across them.

imaginary also optionally supports image placeholder fallback mechanism in case of image processing error or server error of any nature, therefore an image will be always returned by the server in terms of HTTP response body and content MIME type, even in case of error, matching the expected image size and format transparently.

imaginary uses internally libvips, a powerful and efficient library written in C for fast image processing which requires a low memory footprint and it's typically 4x faster than using the quickest ImageMagick and GraphicsMagick settings or Go native image package, and in some cases it's even 8x faster processing JPEG images.

To get started, take a look the installation steps, usage cases and API docs.

Contents

• Supported image operations
• Prerequisites
• Installation
• Docker
• Fly.io
• Cloud Foundry
• Recommended resources
• Production notes
• Scalability
• Clients
• Performance
• Benchmark
• Command-line usage
• HTTP API
• Authorization
• URL signature
• Errors
• Form data
• Params
• Endpoints
• Logging
• Fluentd log ingestion
• Authors
• License

Supported image operations

• Resize
• Enlarge
• Crop
• SmartCrop (based on libvips built-in algorithm)
• Rotate (with auto-rotate based on EXIF orientation)
• AutoRotate with further image transformations (based on EXIF metadata orientation)
• Flip (with auto-flip based on EXIF metadata)
• Flop
• Zoom
• Thumbnail
• Fit
• Pipeline of multiple independent image transformations in a single HTTP request.
• Configurable image area extraction
• Embed/Extend image, supporting multiple modes (white, black, mirror, copy or custom background color)
• Watermark (customizable by text)
• Watermark image
• Custom output color space (RGB, black/white...)
• Format conversion (with additional quality/compression settings)
• Info (image size, format, orientation, alpha...)
• Reply with default or custom placeholder image in case of error.
• Blur

Prerequisites

• libvips 8.8+ (8.9+ recommended)
• C compatible compiler such as gcc 4.6+ or clang 3.0+
• Go 1.12+

Installation

go get -u github.com/h2non/imaginary

Also, be sure you have the latest version of bimg:

go get -u github.com/h2non/bimg

libvips

Run the following script as sudo (supports OSX, Debian/Ubuntu, Redhat, Fedora, Amazon Linux):

curl -s https://raw.githubusercontent.com/h2non/bimg/master/preinstall.sh | sudo bash -

The install script requires curl and pkg-config

Docker

See Dockerfile for image details.

Fetch the image (comes with latest stable Go and libvips versions)

docker pull h2non/imaginary

Start the container with optional flags (default listening on port 9000)

docker run -p 9000:9000 h2non/imaginary -cors -gzip

Start the container enabling remote URL source image processing via GET requests and url query param.

docker run -p 9000:9000 h2non/imaginary imaginary -p 9000 -enable-url-source

Start the container enabling local directory image process via GET requests and file query param.

docker run -p 9000:9000 h2non/imaginary imaginary -p 900 -mount /volume/images

Start the container in debug mode:

docker run -p 9000:9000 -e "DEBUG=*" h2non/imaginary

Enter to the interactive shell in a running container

sudo docker exec -it <containerIdOrName> bash

Stop the container

docker stop h2non/imaginary

For more usage examples, see the command line usage.

All Docker images tags are available here.

Docker Compose

You can add imaginary to your docker-compose.yml file:

version: "3"
services:
  imaginary:
    image: h2non/imaginary:latest
    # optionally mount a volume as local image source
    volumes:
      - images:/mnt/data
    environment:
       PORT: 9000
    command: -enable-url-source -mount /mnt/data
    ports:
      - "9000:9000"

Fly.io

Deploy imaginary in seconds close to your users in Fly.io cloud by clicking on the button below:

About Fly.io

Fly is a platform for applications that need to run globally. It runs your code close to users and scales compute in cities where your app is busiest. Write your code, package it into a Docker image, deploy it to Fly's platform and let that do all the work to keep your app snappy.

You can learn more about how Fly.io can reduce latency and provide a better experience by serving traffic close to your users location.

Global image service tutorial

Learn more about how to run a custom deployment of imaginary on the Fly.io cloud.

CloudFoundry

Assuming you have cloudfoundry account, bluemix or pivotal and command line utility installed.

Clone this repository:

git clone https://github.com/h2non/imaginary.git

Push the application

cf push -b https://github.com/yacloud-io/go-buildpack-imaginary.git imaginary-inst01 --no-start

Define the library path

cf set-env imaginary-inst01 LD_LIBRARY_PATH /home/vcap/app/vendor/vips/lib

Start the application

cf start imaginary-inst01

Recommended resources

Given the multithreaded native nature of Go, in terms of CPUs, most cores means more concurrency and therefore, a better performance can be achieved. From the other hand, in terms of memory, 512MB of RAM is usually enough for small services with low concurrency (<5 requests/second). Up to 2GB for high-load HTTP service processing potentially large images or exposed to an eventual high concurrency.

If you need to expose imaginary as public HTTP server, it's highly recommended to protect the service against DDoS-like attacks. imaginary has built-in support for HTTP concurrency throttle strategy to deal with this in a more convenient way and mitigate possible issues limiting the number of concurrent requests per second and caching the awaiting requests, if necessary.

Production notes

In production focused environments it's highly recommended to enable the HTTP concurrency throttle strategy in your imaginary servers.

The recommended concurrency limit per server to guarantee a good performance is up to 20 requests per second.

You can enable it simply passing a flag to the binary:

$ imaginary -concurrency 20

Graceful shutdown

When you use a cluster, it is necessary to control how the deployment is executed, and it is very useful to finish the containers in a controlled.

You can use the next command:

$ ps auxw | grep 'bin/imaginary' | awk 'NR>1{print buf}{buf = $2}' | xargs kill -TERM > /dev/null 2>&1

Scalability

If you're looking for a large scale solution for massive image processing, you should scale imaginary horizontally, distributing the HTTP load across a pool of imaginary servers.

Assuming that you want to provide a high availability to deal efficiently with, let's say, 100 concurrent req/sec, a good approach would be using a front end balancer (e.g: HAProxy) to delegate the traffic control flow, ensure the quality of service and distribution the HTTP across a pool of servers:

        |==============|
        |  Dark World  |
        |==============|
              ||||
        |==============|
        |   Balancer   |
        |==============|
           |       |
          /         \
         /           \
        /             \
 /-----------\   /-----------\
 | imaginary |   | imaginary | (*n)
 \-----------/   \-----------/

Clients

• node.js/io.js

Feel free to send a PR if you created a client for other language.

Performance

libvips is probably the faster open source solution for image processing. Here you can see some performance test comparisons for multiple scenarios:

• libvips speed and memory usage
• bimg (Go library with C bindings to libvips)

Benchmark

See benchmark.sh for more details

Environment: Go 1.4.2. libvips-7.42.3. OSX i7 2.7Ghz

Requests  [total]       200
Duration  [total, attack, wait]   10.030639787s, 9.949499515s, 81.140272ms
Latencies [mean, 50, 95, 99, max]   83.124471ms, 82.899435ms, 88.948008ms, 95.547765ms, 104.384977ms
Bytes In  [total, mean]     23443800, 117219.00
Bytes Out [total, mean]     175517000, 877585.00
Success   [ratio]       100.00%
Status Codes  [code:count]      200:200

Conclusions

imaginary can deal efficiently with up to 20 request per second running in a multicore machine, where it crops a JPEG image of 5MB and spending per each request less than 100 ms

The most expensive image operation under high concurrency scenarios (> 20 req/sec) is the image enlargement, which requires a considerable amount of math operations to scale the original image. In this kind of operation the required processing time usually grows over the time if you're stressing the server continuously. The advice here is as simple as taking care about the number of concurrent enlarge operations to avoid server performance bottlenecks.

Command-line usage

``` Usage: imaginary -p 80 imaginary -cors imaginary -concurrency 10 imaginary -path-prefix /api/v1 imaginary -enable-url-source imaginary -disable-endpoints form,health,crop,rotate imaginary -enable-url-source -allowed-origins http://localhost,http://server.com,http://*.example.org imaginary -enable-url-source -enable-auth-forwarding imaginary -enable-url-source -authorization "Basic AwDJdL2DbwrD==" imaginary -enable-placeholder imaginary -enable-url-source -placeholder ./placeholder.jpg imaginary -enable-url-signature -url-signature-key 4f46feebafc4b5e988f131c4ff8b5997 imaginary -enable-url-source -forward-headers X-Custom,X-Token imaginary -h | -help imaginary -v | -version

Options: -a Bind address [default: *] -p Bind port [default: 8088] -h, -help Show help -v, -version Show version -path-prefix Url path prefix to listen to [default: "/"] -cors Enable CORS support [default: false] -gzip Enable gzip compression (deprecated) [default: false] -disable-endpoints Comma separated endpoints to disable. E.g: form,crop,rotate,health [default: ""] -key Define API key for authorization -mount