bazel-remote cache

bazel-remote is a HTTP/1.1 and gRPC server that is intended to be used as a remote build cache for REAPI clients like Bazel or as a component of a remote execution service.

The cache contents are stored in a directory on disk with a maximum cache size, and bazel-remote will automatically enforce this limit as needed, by deleting the least recently used files. S3, GCS and experimental Azure blob storage proxy backends are also supported.

Note that while bazel-remote is consumable as a go module, we provide no guarantees on the stability or backwards compatibility of the APIs. We do attempt to keep the standalone executable backwards-compatible between releases however, and cache directory format changes are only allowed in major version upgrades.

Project status: bazel-remote has been serving TBs of cache artifacts per day since April 2018, both on commodity hardware and AWS servers. Outgoing bandwidth can exceed 15 Gbit/s on the right AWS instance type.

HTTP/1.1 REST API

Cache entries are set and retrieved by key, and there are two types of keys that can be used: 1. Content addressed storage (CAS), where the key is the lowercase SHA256 hash of the entry. The REST API for these entries is: /cas/<key> or with an optional but ignored instance name: /<instance>/cas/<key>. 2. Action cache, where the key is an arbitrary 64 character lowercase hexadecimal string. Bazel uses the SHA256 hash of an action as the key, to store the metadata created by the action. The REST API for these entries is: /ac/<key> or with an optional instance name: /<instance>/ac/<key>.

Values are stored via HTTP PUT requests, and retrieved via GET requests. HEAD requests can be used to confirm whether a key exists or not.

If GET requests specify zstd in the Accept-Encoding header, then zstandard-encoded data may be returned.

To upload zstandard compressed data, PUT requests must set Content-Encoding: zstd and include a custom X-Digest-SizeBytes header with the size of the uncompressed entry. The key must also refer to the uncompressed entry.

If the --enable_ac_key_instance_mangling flag is specified and the instance name is not empty, then action cache keys are hashed along with the instance name to produce the action cache lookup key. Since the URL path is processed with Go's path.Clean function before extracting the instance name, clients should avoid using repeated slashes, ./ and ../ in the URL.

Values stored in the action cache are validated as an ActionResult protobuf message as per the Bazel Remote Execution API v2 unless validation is disabled by configuration. The HTTP server also supports reading and writing JSON encoded protobuf ActionResult messages to the action cache by using HTTP headers Accept: application/json for GET requests and Content-type: application/json for PUT requests.

Useful endpoints

/status

Returns the cache status/info.

$ curl http://localhost:8080/status
{
 "CurrSize": 414081715503,
 "ReservedSize": 876400,
 "MaxSize": 8589934592000,
 "NumFiles": 621413,
 "ServerTime": 1746258977,
 "GitCommit": "d0f166cdd973342ec4aa8a51228cfd3a7a205414",
 "GitTags": "v2.5.1",
 "NumGoroutines": 12
}

/cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The empty CAS blob is always available, even if the cache is empty. This can be used to test that a bazel-remote instance is running and accepting requests.

$ curl --head --fail http://localhost:8080/cas/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
HTTP/1.1 200 OK
Content-Length: 0
Date: Fri, 01 May 2020 10:42:06 GMT

Prometheus Metrics

To query endpoint metrics see github.com/slok/go-http-metrics's query examples.

gRPC API

bazel-remote also supports the ActionCache, ContentAddressableStorage and Capabilities services in the Bazel Remote Execution API v2, and the corresponding parts of the Byte Stream API.

When using the --enable_ac_key_instance_mangling feature, clients are advised to avoid repeated slashes, ../ and ./ strings in the instance name, for consistency with the HTTP interface.

Prometheus Metrics

To query endpoint metrics see github.com/grpc-ecosystem/go-grpc-prometheus's metrics documentation.

Experimental Remote Asset API Support

There is (very) experimental support for a subset of the Fetch service in the Remote Asset API which can be enabled with the --experimental_remote_asset_api flag.

To use this with Bazel, specify --experimental_remote_downloader=grpc://replace-with-your.host:port.

Byte Stream compressed-blobs

This version of bazel-remote supports the Byte Stream compressed-blobs REAPI feature, which provides a way for clients to upload and download CAS blobs compressed with zstandard, in order to improve network efficiency.

Uploaded CAS blobs are stored in a zstandard compressed format by default, which can increase the effective cache size and reduce load on the server if clients also download blobs in zstandard compressed form. If you would rather store CAS blobs in uncompressed form, add --storage_mode uncompressed to your configuration.

Usage

If a YAML configuration file is specified by the --config_file command line flag or BAZEL_REMOTE_CONFIG_FILE environment variable, then other command line flags and environment variables are ignored. Otherwise, the flags and environment variables listed in the help text below can be specified (flags override the corresponding environment variables).

See examples/bazel-remote.service for an example (systemd) linux setup.

Command line flags

``` $ ./bazel-remote --help bazel-remote - A remote build cache for Bazel and other REAPI clients

USAGE: bazel-remote [options]

OPTIONS: --config_file value Path to a YAML configuration file. If this flag is specified then all other flags are ignored. [$BAZEL_REMOTE_CONFIG_FILE]

--dir value Directory path where to store the cache contents. This flag is required. [$BAZEL_REMOTE_DIR]

--max_size value The maximum size of bazel-remote's disk cache in GiB. This flag is required. (default: 0) [$BAZEL_REMOTE_MAX_SIZE]

--storage_mode value Which format to store CAS blobs in. Must be one of "zstd" or "uncompressed". (default: "zstd") [$BAZEL_REMOTE_STORAGE_MODE]

--zstd_implementation value ZSTD implementation to use. Supported values: "cgo", "go" (default: "go") [$BAZEL_REMOTE_ZSTD_IMPLEMENTATION]

--http_address value Address specification for the HTTP server listener, formatted either as [host]:port for TCP or unix://path.sock for Unix domain sockets. [$BAZEL_REMOTE_HTTP_ADDRESS]

--host value DEPRECATED. Use --http_address to specify the HTTP server listener. [$BAZEL_REMOTE_HOST]

--port value DEPRECATED. Use --http_address to specify the HTTP server listener. (default: 8080) [$BAZEL_REMOTE_PORT]

--grpc_address value Address specification for the gRPC server listener, formatted either as [host]:port for TCP or unix://path.sock for Unix domain sockets. Set to 'none' to disable. [$BAZEL_REMOTE_GRPC_ADDRESS]

--grpc_port value DEPRECATED. Use --grpc_address to specify the gRPC server listener. Set to 0 to disable. (default: 9092) [$BAZEL_REMOTE_GRPC_PORT]

--profile_address value Address specification for a http server to listen on for profiling, formatted either as [host]:port for TCP or unix://path.sock for Unix domain sockets. Off by default, but can also be set to 'none' to disable explicitly. (default: "", ie profiling disabled) [$BAZEL_REMOTE_PROFILE_ADDRESS]

--profile_host value DEPRECATED. Use --profile_address instead. A host address to listen on for profiling, if enabled by a valid --profile_port setting. (default: "127.0.0.1") [$BAZEL_REMOTE_PROFILE_HOST]

--profile_port value DEPRECATED. Use --profile_address instead. If a positive integer, serve /debug/pprof/* URLs from http://profile_host:profile_port. (default: 0, ie profiling disabled) [$BAZEL_REMOTE_PROFILE_PORT]

--http_read_timeout value The HTTP read timeout for a client request in seconds (does not apply to the proxy backends or the profiling endpoint) (default: 0s, ie disabled) [$BAZEL_REMOTE_HTTP_READ_TIMEOUT]

--http_write_timeout value The HTTP write timeout for a server response in seconds (does not apply to the proxy backends or the profiling endpoint) (default: 0s, ie disabled) [$BAZEL_REMOTE_HTTP_WRITE_TIMEOUT]

--htpasswd_file value Path to a .htpasswd file. This flag is optional. Please read https://httpd.apache.org/docs/2.4/programs/htpasswd.html. [$BAZEL_REMOTE_HTPASSWD_FILE]

--min_tls_version value The minimum TLS version that is acceptable for incoming requests (does not apply to proxy backends). Allowed values: 1.0, 1.1, 1.2, 1.3. (default: "1.0") [$BAZEL_REMOTE_MIN_TLS_VERSION]

--tls_ca_file value Optional. Enables mTLS (authenticating client certificates), should be the certificate authority that signed the client certificates. [$BAZEL_REMOTE_TLS_CA_FILE]

--tls_cert_file value Path to a pem encoded certificate file. [$BAZEL_REMOTE_TLS_CERT_FILE]

--tls_key_file value Path to a pem encoded key file. [$BAZEL_REMOTE_TLS_KEY_FILE]

--allow_unauthenticated_reads If authentication is enabled (--htpasswd_file or --tls_ca_file), allow unauthenticated clients read access. (default: false, ie if authentication is required, read-only requests must also be authenticated) [$BAZEL_REMOTE_UNAUTHENTICATED_READS]

--idle_timeout value The maximum period of having received no request after which the server will shut itself down. (default: 0s, ie disabled) [$BAZEL_REMOTE_IDLE_TIMEOUT]

--max_queued_uploads value When using proxy backends, sets the maximum number of objects in queue for upload. If the queue is full, uploads will be skipped until the queue has space again. (default: 1000000) [$BAZEL_REMOTE_MAX_QUEUED_UPLOADS]

--max_blob_size value The maximum logical/uncompressed blob size that will be accepted from clients. Note that this limit is not applied to preexisting blobs in the cache. (default: 9223372036854775807) [$BAZEL_REMOTE_MAX_BLOB_SIZE]

--max_proxy_blob_size value The maximum logical/uncompressed blob size that will be downloaded from proxies. Note that this limit is not applied to preexisting blobs in the cache. (default: 9223372036854775807) [$BAZEL_REMOTE_MAX_PROXY_BLOB_SIZE]

--num_uploaders value When using proxy backends, sets the number of Goroutines to process parallel uploads to backend. (default: 100) [$BAZEL_REMOTE_NUM_UPLOADERS]

--grpc_proxy.url value The base URL to use for the experimental grpc proxy backend, e.g. grpc://localhost:9090 or grpcs://example.com:7070. Note that this requires a backend with remote asset API support if you want http client requests to work. [$BAZEL_REMOTE_GRPC_PROXY_URL]

--grpc_proxy.key_file value Path to a key used to authenticate with the proxy backend using mTLS. If this flag is provided, then grpc_proxy.cert_file must also be specified. [$BAZEL_REMOTE_GRPC_PROXY_KEY_FILE]

--grpc_proxy.cert_file value Path to a certificate used to authenticate with the proxy backend using mTLS. If this flag is provided, then grpc_proxy.key_file must also be specified. [$BAZEL_REMOTE_GRPC_PROXY_CERT_FILE]

--grpc_proxy.ca_file value Path to a certificate autority used to validate the grpc proxy backend certificate. [$BAZEL_REMOTE_GRPC_PROXY_CA_FILE]

--http_proxy.url value The base URL to use for a http proxy backend. [$BAZEL_REMOTE_HTTP_PROXY_URL]

--http_proxy.key_file value Path to a key used to authenticate with the proxy backend using mTLS. If this flag is provided, then http_proxy.cert_file must also be specified. [$BAZEL_REMOTE_HTTP_PROXY_KEY_FILE]

--http_proxy.cert_file value Path to a certificate used to authenticate with the proxy backend using mTLS. If this flag is provided, then http_proxy.key_file must also be specified. [$BAZEL_REMOTE_HTTP_PROXY_CERT_FILE]

--http_proxy.ca_file value Path to a certificate autority used to validate the http proxy backend certificate. [$BAZEL_REMOTE_HTTP_PROXY_CA_FILE]

--gcs_proxy.bucket value The bucket to use for the Google Cloud Storage proxy backend. [$BAZEL_REMOTE_GCS_BUCKET]

--gcs_proxy.use_default_credentials Whether or not to use authentication for the Google Cloud Storage proxy backend. (default: false) [$BAZEL_REMOTE_GCS_USE_DEFAULT_CREDENTIALS]

--gcs_proxy.json_credentials_file value Path to a JSON file that contains Google credentials for the Google Cloud Storage proxy backend. [$BAZEL_REMOTE_GCS_JSON_CREDENTIALS_FILE]

--ldap.url va