image-steam

A simple, fast, and highly customizable realtime image manipulation web server built atop Node.js. Serving (many) millions of images daily by way of simple (single DC, simple storage) and advanced (multi-DC geo-distributed, replicated storage, multi-tier caching) configurations. Responsible in part for this white paper.

Notice: v0.41 Introduces Minor Breaking Changes

What is Image Steam?

http://localhost:13337/my-app/some-path/my-image.jpg/:/rs=w:640/cr=l:5%25,t:10%25,w:90%25,h:80%25/fx-gs

Consider the above isteam example, by allowing clients to directly request any variation of an image using few or many image instructions, and in near realtime. This opens the doors to improved user and developer experiences. The above example takes the requested image some-path/my-image.jpg, resizes to a fixed width (preserving aspect, by default), crops around the edges like a picture frame, applies greyscale effects, and will auto-select the most optimal image format supported by the requesting device -- all without developer support, and in (near) realtime.

Layer	Info
node.js http(s)	RESTful interface
routing	URI pathing that make up the desired image operations. Device aware - Supports implied optimizations based on the given request (notably webp support)
throttling	Quality of service management
security	Allows protecting resources behind the tranformations via signed requests
optimized original	Optimization of uploaded originals
storage	Storage engine for reads (original or cache) and writes (cache). Extensible storage clients for storage and caching
processor	Image operations (resize, crop, etc) go here, powered by Sharp and libvips

Why Image Steam?

There are a number of options out there, but isteam differentiates itself by:

Feature	Info
Highly configurable	Everything all the way down to how image operations are mapped can be overridden. Most solutions are very prescriptive on how it must work. isteam is intended to adhere to your architecture, your storage, your caching, your replication patterns
Optimized	Optimizes originally uploaded asset to account for large uploads, enabling a higher quality service by making the pipeline for image operations substantially faster. A critical feature in supporting large media of todays modern devices
QoS	Quality of service features such as throttling and memory thresholds, to best take advantage of your hardware under ideal and non ideal scenarios
Device Aware	Device centric responses, where more than a URI may influence response. Compression and Accepts header (i.e. WebP) being examples
CDN Support	Supported, but not required
Library Agnostic	Provides an abstraction atop image processing libraries, enabling per-operation level of control to enable using the right tool for the given operation. Bugs, features, performance are a few of the factors that may influence this
Friendly CLI	No custom app required to create your image API
Platform Agnostic	Good *Nix & Windows support

Installation

The speed and power of this module would not be possible without the incredible work of libvips (low level image processor), Sharp (depends on libvips), and xxHash for lightning-fast hashing.

Install Sharp via http://sharp.dimens.io/en/stable/install/ - This should also take care of the libvips dependency.

Usage

Standalone Server

While Routing, Throttling, and Storage are all independently usable and configurable, a basic usage example that pulls everything together can be as simple as:

npm install image-steam -g
isteam --isConfig './myconfig.json' --isDefaults './mydefaults.json'

Defaults are optional. Config can also be a CommonJS file:

isteam --isConfig './myconfig.js'

Connect Middleware

Or if you prefer to incorporate into your own app:

var http = require('http');
var imgSteam = require('image-steam');

http.createServer(new imgSteam.http.Connect({ /* options */ }).getHandler())
  .listen(13337, '127.0.0.1')
;

Which is equivalent of cloning this repo and invoking npm start.

Cloning

If you plan to hack on isteam:

git clone git@github.com:asilvas/node-image-steam.git
cd ./node-image-steam
npm install
npm start

^ May require sudo or admin rights.

Performance

While this module provides granular control over HTTP throttling to provide the highest quality of service possible, performance is entirely from Sharp and libvips: http://sharp.dimens.io/en/stable/performance/#performance

Options

isteam --isConfig './myconfig.json'

HTTP Options

{
  "http": [
    { "port": 80 },
    { "port": 443, "ssl": {} }
  ]
}

Option	Type	Default	Info
port	number	13337	Port to bind to
host	string	"localhost"	Host to bind to
backlog	number	511	TCP backlog
ssl	TLS Options	undefined	If object provided, will bind with TLS Options
ssl.pfx	Buffer or string	undefined	If string, will auto-load from file system
ssl.key	Buffer or string	undefined	If string, will auto-load from file system
ssl.cert	Buffer or string	undefined	If string, will auto-load from file system
globalAgent	AgentKeepAlive	Defaults	If object, will initialize the global http(s) agents

Storage Options

{
  "storage": {
    "defaults": {
      "driverPath": "image-steam-s3",
      "endpoint": "s3.amazonaws.com",
      "accessKey": "abc",
      "secretKey": "123"
    }
  }
}

Option	Type	Default	Info
defaults	StorageOptions	optional	If provided, these options will be the defaults applied to all StorageOptions. Note: At least one of defaults, app, domain, or header are required
app	Object<StorageOptions>	optional	If provided, allows for driver-specific options to be applied on a per-request basis, based on the route. If no match is found, the original options provided at initialization will be used. Example: { "some-app": StorageOptions } }.
domain	Object<StorageOptions>	optional	If provided, allows for driver-specific options to be applied on a per-request basis, based on the host header. If no match is found, the original options provided at initialization will be used. Example: { "somedomain.com": StorageOptions }. Note: You must still provide root level storage options to act as defaults
header	Object<StorageOptions>	optional	If provided, allows for driver-specific options to be applied on a per-request basis, based on x-isteam-app header. If no match is found, the original options provided at initialization will be used. Example: { "some-other-app": StorageOptions }. Note: You must still provide root level storage options to act as defaults
cache	StorageOptions	optional	If provided, allows for driver-specific options to be applied for all cache objects. This effectively puts the api into a read-only mode for original assets, with all writes going exclusively to a single cache store
cacheOptimized	StorageOptions	optional	If provided, will attempt to access only optimized original assets from this storage. This permits splitting of cache for sake of replication or eviction policies. If supplied, this also means delete requests will be supplied to both caches
cacheArtifacts	boolean	true	If false, processed image artifacts will not be cached. Useful if limited cache capacity
cacheTTS	number	optional	If provided, when artifacts or optimized originals (unless cacheOptimizedTTS is also provided) are fetched from cache, if the age of the object exceeds this time-to-stale value (in seconds), it's age will be reset (implementation varies by storage client, but defaults to copying onto itself). This is a powerful pattern in cases where the cache storage leverages time-to-live, but you do not want active objects to be deleted at the expense of the user experience (and cost). When an object is "refreshed", it will only impact the storage of the stale object, ignoring replicas option. A refresh is out-of-band of the request
cacheOptimizedTTS	number	optional	If provided, when optimized original are fetched from cache, if the age of the object exceeds this time-to-stale value (in seconds), it's age will be reset (implementation varies by storage client, but defaults to copying onto itself). This is a powerful pattern in cases where the cache storage leverages time-to-live, but you do not want active objects to be deleted at the expense of the user experience (and cost). When an object is "refreshed", it will only impact the storage of the stale object, ignoring replicas option. A refresh is out-of-band of the request
replicas	Object<StorageReplica>	optional	If provided, all cache writes will also be written (out-of-band) to the desired storage replicas. Example: { remoteCache: { cache: { /* options */ }, cacheOptimized: { /* options */ } } }. Where remoteCache is the name of a storage I want to forward my writes to. This feature provides a high degree of flexibility when determining your distribution of data across the globe, without the fixed replication that may be permitted by the storage provided (ala S3 replication).
replicas[].cache	StorageOptions	optional	Same behavior as storage.cache
replicas[].cacheOptimized	StorageOptions	optional	Same behavior as storage.cacheOptimized
replicas[].replicateArtifacts	boolean	true	In some cases it may be too costly to replicate all image artifacts, especially when the location you're replicating to may receive small amounts of traffic for the same images. By disabling this flag, only optimized original images will be written to replicas

StorageOptions

Option	Type	Default	Info
driver	string	driver or driverPath required	Bundled storage driver to use
driverPath	string	driver or driverPath required	Load a custom driver from the desired path, ignoring the driver option
(driver options)			All other options will be supplied to the storage driver indicated by driver or driverPath

Advanced storage example:

{
  "storage": {
    "defaults": {
      "driverPath": "image-steam-s3",
      "endpoint": "s3.amazonaws.com
    },
    "app": {
      "app1": {
        "accessKey": "key1",
        "secretKey": "secret1"
      }
    },
    "cache": {
      "endpoint": "<dc1 endpoint>",
      "accessKey": "key2",
      "secretKey": "secret2"
    },
    "cacheTTS": 86400, /* 24 hrs */
    "cacheOptimizedTTS": 43200, /* 12 hrs */
    "replicas": {
      "dc2Cache": {
        "cache": {
          "endpoint": "<dc2 endpoint>",
          "accessKey": "key3",
          "secretKey": "secret3"
        }
      }
    }
  }
}

Bundled Storage Clients

Storage Client - File System

Option	Type	Default	Info
driver=fs			File System
path	string	required	Root path on file system

Storage Client - HTTP

Option	Type	Default	Info
driver=http			Read-Only driver for web resource
endpoint	string	required	Endpoint of http(s) service
bucket	string	optional	If provided, will not attempt to take bucket from path

Custom Client

Custom storage types can easily be added via exporting fetch and store. See lib/storage/fs (read/write/delete) or lib/storage/http (read only) for reference.

External Storage Clients

• https://github.com/asilvas/image-steam-aws-s3 - S3 client built on AWS S3
• https://github.com/asilvas/image-steam-s3 - S3 client built on knox
• https://github.com/asilvas/image-steam-redis - Redis client built on ioredis
• https://github.com/asilvas/image-steam-blobby - Blobby client over HTTP(S)

Throttle Options

Throttling allows for fine grain control over quality of service, as well as optimizing to your hardware.

{
  "throttle": {
    "ccProcessors": 4,
    "ccPrefetchers": 20,
    "ccRequests": 100
  }
}

Option	Type	Default	Info
ccProcessors	number	4	Number of concurrent image processing operations. Anything to exceed this value will wait (via semaphore) for next availability
ccPrefetchers	number	20	Number of concurrent storage request operations. This helps prevent saturation of your storage and/or networking interfaces to provide the optimal experience
ccRequests	number	100	Number of concurrent http requests. Anything to exceed this value will result in a 503 (too b