Bottlerocket Update Operator

The Bottlerocket update operator (or, for short, Brupop) is a Kubernetes operator that coordinates Bottlerocket updates on hosts in a cluster. When installed, the Bottlerocket update operator starts a controller deployment on one node, an agent daemon set on every Bottlerocket node, and an Update Operator API Server deployment. The controller orchestrates updates across your cluster, while the agent is responsible for periodically querying for Bottlerocket updates, draining the node, and performing the update when asked by the controller. The agent performs all cluster object mutation operations via the API Server, which performs additional authorization using the Kubernetes TokenReview API -- ensuring that any request associated with a node is being made by the agent pod running on that node. Further, cert-manager is required in order for the API server to use a CA certificate to communicate over SSL with the agents. Updates to Bottlerocket are rolled out in waves to reduce the impact of issues; the nodes in your cluster may not all see updates at the same time.

For a deep dive on installing Brupop, how it works, and its integration with Bottlerocket, check out this design deep dive document!

Getting Started

Version Support Policy

As per our policy, Brupop follows the semantic versioning (semver) principles, ensuring that any updates in minor versions do not introduce any breaking or backward-incompatible changes. However, please note that we only provide security patches for the latest minor version. Therefore, it is highly recommended to always keep your Brupop installation up to date with the latest available version.

For example: If v1.3.0 is the latest Brupop release, then, v1.3 (latest minor version) will be considered as supported and v1.3.0 (latest available version) will be the recommended version of Brupop to be installed. When v1.3.1 is released, then that version will be considered as recommended.

Installation

1. Brupop uses cert-manager to manage self-signed certificates used by the Bottlerocket update operator. To install cert-manager:

kubectl apply -f \
  https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml

Or if you're using helm:

# Add the cert-manager helm chart
helm repo add jetstack https://charts.jetstack.io

# Update your local chart cache with the latest
helm repo update

# Install the cert-manager (including its CRDs)
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.8.2 \
  --set installCRDs=true

1. The Bottlerocket update operator can then be installed using helm:

# Add the bottlerocket-update-operator chart
helm repo add brupop https://bottlerocket-os.github.io/bottlerocket-update-operator

# Update your local chart cache with the latest updates
helm repo update

# Create a namespace
kubectl create namespace brupop-bottlerocket-aws

# Install the brupop CRD
helm install brupop-crd brupop/bottlerocket-shadow

# Install the brupop operator
helm install brupop-operator brupop/bottlerocket-update-operator

This will create the custom resource definition, roles, deployments, etc., and use the latest update operator image available in Amazon ECR Public.

Alternatively, you can use the pre-baked manifest, with all the default values, found at the root of this repository (named bottlerocket-update-operator.yaml). This YAML manifest file is also attached to each release found in the GitHub releases page.

Note: The generated manifests points to the latest version of the Update Operator, v1.1.0. Be sure to use the release for the Update Operator release that you plan to deploy.

1. Label nodes with bottlerocket.aws/updater-interface-version=2.0.0 to indicate they should be automatically updated. Only bottlerocket nodes with this label will be updated. For more information on labeling, refer to the Label nodes section of this readme.

kubectl label node MY_NODE_NAME bottlerocket.aws/updater-interface-version=2.0.0

Configuration

Configure via Helm values yaml file

You can use a values file when installing brupop with helm (via the --values / -f flag) to configure how brupop functions:

# Default values for bottlerocket-update-operator.

# The namespace to deploy the update operator into
namespace: "brupop-bottlerocket-aws"

# The image to use for brupop
# This defaults to the image built alongside a particular helm chart, but you can override it by uncommenting this line:
# image: "public.ecr.aws/bottlerocket/bottlerocket-update-operator:v1.8.0"

# Placement controls
# See the Kubernetes documentation about placement controls for more details:
# * https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
# * https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector
# * https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
placement:
  agent:
    # The agent is a daemonset, so the only controls that apply to it are tolerations.
    tolerations: []

  controller:
    tolerations: []
    nodeSelector: {}
    podAffinity: {}
    podAntiAffinity: {}

  apiserver:
    tolerations: []
    nodeSelector: {}
    podAffinity: {}
    # By default, apiserver pods prefer not to be scheduled to the same node.
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: brupop.bottlerocket.aws/component
                operator: In
                values:
                - apiserver
            topologyKey: kubernetes.io/hostname

# If testing against a private image registry, you can set the pull secret to fetch images.
# This can likely remain as `brupop` so long as you run something like the following:
# kubectl create secret docker-registry brupop \
#  --docker-server 109276217309.dkr.ecr.us-west-2.amazonaws.com \
#  --docker-username=AWS \
#  --docker-password=$(aws --region us-west-2 ecr get-login-password) \
#  --namespace=brupop-bottlerocket-aws
#imagePullSecrets:
#  - name: "brupop"

# External load balancer setting.
# When `exclude_from_lb_wait_time_in_sec` is set to a positive value
# brupop will exclude the node from load balancing
# and will wait for `exclude_from_lb_wait_time_in_sec` seconds before draining nodes.
# Under the hood, this uses the `node.kubernetes.io/exclude-from-external-load-balancers` label
# to exclude those nodes from load balancing.
exclude_from_lb_wait_time_in_sec: "0"

# Concurrent update nodes setting.
# When `max_concurrent_updates` is set to a positive integer value,
# brupop will concurrently update max `max_concurrent_updates` nodes.
# When `max_concurrent_updates` is set to "unlimited",
# brupop will concurrently update all nodes with respecting `PodDisruptionBudgets`
# Note: the "unlimited" option does not work well with `exclude_from_lb_wait_time_in_sec`
# option, which could potential exclude all nodes from load balancer at the same time.
max_concurrent_updates: "1"

# DEPRECATED: use the scheduler settings
# Start and stop times for update window
# Brupop will operate node updates within update time window.
# when you set up time window start and stop time, you should use UTC (24-hour time notation).
update_window_start: "0:0:0"
update_window_stop: "0:0:0"

# Scheduler setting
# Brupop will operate node updates on scheduled maintenance windows by using cron expressions.
# When you set up the scheduler, you should follow cron expression rules.
# ┌───────────── seconds (0 - 59)
# │ ┌───────────── minute (0 - 59)
# │ │ ┌───────────── hour (0 - 23)
# │ │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ │ ┌───────────── month (Jan, Feb, Mar, Apr, Jun, Jul, Aug, Sep, Oct, Nov, Dec)
# │ │ │ │ │ ┌───────────── day of the week (Mon, Tue, Wed, Thu, Fri, Sat, Sun)
# │ │ │ │ │ │ ┌───────────── year (formatted as YYYY)
# │ │ │ │ │ │ │
# │ │ │ │ │ │ │
# * * * * * * *
scheduler_cron_expression: "* * * * * * *"

# API server ports
# The port the API server uses for its own operations. This is accessed by the controller,
# the bottlerocket-shadow daemonset, etc.
apiserver_internal_port: "8443"
# API server internal address where the CRD version conversion webhook is served
apiserver_service_port: "443"

logging:
  # Formatter for the logs emitted by brupop.
  # Options are:
  # * full - Human-readable, single-line logs
  # * compact - A variant of full optimized for shorter line lengths
  # * pretty - "Excessively pretty" logs optimized for human-readable terminal output.
  # * json - Newline-delimited JSON-formatted logs.
  formatter: "pretty"
  # Whether or not to enable ANSI colors on log messages.
  # Makes the output "pretty" in terminals, but may add noise to web-based log utilities.
  ansi_enabled: "true"

  # Controls the filter for tracing/log messages.
  # This can be as simple as a log-level (e.g. "info", "debug", "error"), but also supports more complex directives.
  # See https://docs.rs/tracing-subscriber/0.3.17/tracing_subscriber/filter/struct.EnvFilter.html#directives
  controller:
    tracing_filter: "info"
  agent:
    tracing_filter: "info"
  apiserver:
    tracing_filter: "info"

# Provide pod level labels for the brupop resources.
podLabels: {}

Configuration via Kubernetes yaml

Configure API server ports

If you'd like to configure what ports the API server uses, adjust the value that is consumed in the container environment:

      ...
      containers:
        - command:
            - "./api-server"
          env:
            - name: APISERVER_INTERNAL_PORT
              value: 999

You'll then also need to adjust the various "port" entries in the YAML manifest to correctly reflect what port the API server starts on and expects for its service port:

    ...
    webhook:
      clientConfig:
        service:
          name: brupop-apiserver
          namespace: brupop-bottlerocket-aws
          path: /crdconvert
          port: 123

The default values are generated from the default values yaml file file: - apiserver_internal_port: "8443" - This is the container port the API server starts on. If this environment variable is not found, the Brupop API server will fail to start. - apiserver_service_port: "443" - This is the port Brupop's Kubernetes Service uses to target the internal API Server port. It is used by the node agents to access the API server. If this environment variable is not found, the Brupop agents will fail to start.

Resource Requests & Limits

The bottlerocket-update-operator.yaml manifest makes several default recommendations for Kubernetes resource requests and limits. In general, the update operator and its components are lite-weight and shouldn't consume more than 10m CPU (which is roughly equivalent to 1/100th of a CPU core) and 50Mi (which is roughly equivalent to 0.05 GB of memory). If this limit is breached, the Kubernetes API will restart the faulting container.

Note that your mileage with these resource requests and limits may vary. Any number of factors may contribute to varying results in resource utilization (different compute instance types, workload utilization, API ingress/egress, etc). The Kubernetes documentation for Resource Management of Pods and Containers is an excellent resource for understanding how various compute resources are utilized and how Kubernetes manages these resources.

If resource utilization by the brupop components is not a concern, removing the resources fields in the manifest will not affect the functionality of any components.

Exclude Nodes from Load Balancers Before Draining

This configuration uses Kubernetes ServiceNodeExclusion feature. EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC can be used to enable the feature to exclude the node from load balancer before draining. When EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC is 0 (default), the feature is disabled. When EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC is set to a positive integer, bottlerocket update operator will exclude the node from load balancer and then wait EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC seconds before draining the pods on the node.

To enable this feature, set the exclude_from_lb_wait_time_in_sec value in your helm values yaml file to a positive integer. For example, exclude_from_lb_wait_time_in_sec: "100".

Otherwise, go to bottlerocket-update-operator.yaml and change EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC to a positive integer value. For example:

      ...
      containers:
        - command:
            - "./agent"
          env:
            - name: MY_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC
              value: "180"
      ...

Set Up Max Concurrent Update

MAX_CONCURRENT_UPDATE can be used to specify the max concurrent updates during updating. When MAX_CONCURRENT_UPDATE is a positive integer, the bottlerocket update operator will concurrently update u