mongobetween

mongobetween is a lightweight MongoDB connection pooler written in Golang. It's primary function is to handle a large number of incoming connections, and multiplex them across a smaller connection pool to one or more MongoDB clusters.

mongobetween is used in production at Coinbase. It is currently deployed as a Docker sidecar alongside a Rails application using the Ruby Mongo driver, connecting to a number of sharded MongoDB clusters. It was designed to connect to mongos routers who are responsible for server selection for read/write preferences (connecting directly to a replica set's mongod instances hasn't been battle tested).

How it works

mongobetween listens for incoming connections from an application, and proxies any queries to the MongoDB Go driver which is connected to a MongoDB cluster. It also intercepts any ismaster commands from the application, and responds with "I'm a shard router (mongos)", without proxying. This means mongobetween appears to the application as an always-available MongoDB shard router, and any MongoDB connection issues or failovers are handled internally by the Go driver.

Installation

go install github.com/coinbase/mongobetween

Usage

Usage: mongobetween [OPTIONS] address1=uri1 [address2=uri2] ...
  -loglevel string
        One of: debug, info, warn, error, dpanic, panic, fatal (default "info")
  -network string
        One of: tcp, tcp4, tcp6, unix or unixpacket (default "tcp4")
  -password string
        MongoDB password
  -ping
        Ping downstream MongoDB before listening
  -pretty
        Pretty print logging
  -statsd string
        Statsd address (default "localhost:8125")
  -unlink
        Unlink existing unix sockets before listening
  -username string
        MongoDB username
  -dynamic string
        File or URL to query for dynamic configuration
  -enable-sdam-metrics
        Enable SDAM(Server Discovery And Monitoring) metrics
  -enable-sdam-logging
        Enable SDAM(Server Discovery And Monitoring) logging

TCP socket example:

mongobetween ":27016=mongodb+srv://username:password@cluster.mongodb.net/database?maxpoolsize=10&label=cluster0"

Unix socket example:

mongobetween -network unix "/tmp/mongo.sock=mongodb+srv://username:password@cluster.mongodb.net/database?maxpoolsize=10&label=cluster0"

Proxying multiple clusters:

mongobetween -network unix \
  "/tmp/mongo1.sock=mongodb+srv://username:password@cluster1.mongodb.net/database?maxpoolsize=10&label=cluster1" \
  "/tmp/mongo2.sock=mongodb+srv://username:password@cluster2.mongodb.net/database?maxpoolsize=10&label=cluster2"

The label query parameter in the connection URI is used to any tag statsd metrics or logs for that connection.

Dynamic configuration

Passing a file or URL as the -dynamic argument will allow somewhat dynamic configuration of mongobetween. Example supported file format:

{
  "Clusters": {
    ":12345": {
      "DisableWrites": true,
      "RedirectTo": ""
    },
    "/var/tmp/cluster1.sock": {
      "DisableWrites": false,
      "RedirectTo": "/var/tmp/cluster2.sock"
    }
  }
}

This will disable writes to the proxy served from address :12345, and redirect any traffic sent to /var/tmp/cluster1.sock to the proxy running on /var/tmp/cluster2.sock. This is useful for minimal-downtime migrations between clusters.

TODO

Current known missing features: - [X] Transaction server pinning - [X] Different cursors on separate servers with the same cursor ID value

Statsd

mongobetween supports reporting health metrics to a local statsd sidecar, using the Datadog Go library. By default it reports to localhost:8125. The following metrics are reported: - mongobetween.handle_message (Timing) - end-to-end time handling an incoming message from the application - mongobetween.round_trip (Timing) - round trip time sending a request and receiving a response from MongoDB - mongobetween.request_size (Distribution) - request size to MongoDB - mongobetween.response_size (Distribution) - response size from MongoDB - mongobetween.open_connections (Gauge) - number of open connections between the proxy and the application - mongobetween.connection_opened (Counter) - connection opened with the application - mongobetween.connection_closed (Counter) - connection closed with the application - mongobetween.cursors (Gauge) - number of open cursors being tracked (for cursor -> server mapping) - mongobetween.transactions (Gauge) - number of transactions being tracked (for client sessions -> server mapping)**** - mongobetween.server_selection (Timing) - Go driver server selection timing - mongobetween.checkout_connection (Timing) - Go driver connection checkout timing - mongobetween.pool.checked_out_connections (Gauge) - number of connections checked out from the Go driver connection pool - mongobetween.pool.open_connections (Gauge) - number of open connections from the Go driver to MongoDB - mongobetween.pool_event.connection_closed (Counter) - Go driver connection closed - mongobetween.pool_event.connection_pool_created (Counter) - Go driver connection pool created - mongobetween.pool_event.connection_created (Counter) - Go driver connection created - mongobetween.pool_event.connection_check_out_failed (Counter) - Go driver connection check out failed - mongobetween.pool_event.connection_checked_out (Counter) - Go driver connection checked out - mongobetween.pool_event.connection_checked_in (Counter) - Go driver connection checked in - mongobetween.pool_event.connection_pool_cleared (Counter) - Go driver connection pool cleared - mongobetween.pool_event.connection_pool_closed (Counter) - Go driver connection pool closed

Background

mongobetween was built to address a connection storm issue between a high scale Rails app and MongoDB (see blog post). Due to Ruby MRI's global interpreter lock, multi-threaded web applications don't utilize multiple CPU cores. To achieve better CPU utilization, Puma is run with multiple workers (processes), each of which need a separate MongoDB connection pool. This leads to a large number of connections to MongoDB, sometimes exceeding MongoDB's upstream connection limit of 128k connections.

mongobetween has reduced connection counts by an order of magnitude, spikes of up to 30k connections are now reduced to around 2k. It has also significantly reduced ismaster commands on the cluster, as there's only a single monitor goroutine per mongobetween process, instead of a monitor thread for each Ruby process.