Isopod

Isopod is an expressive DSL framework for Kubernetes configuration. Without intermediate YAML artifacts, Isopod renders Kubernetes objects as Protocol Buffers, so they are strongly typed and consumed directly by the Kubernetes API.

With Isopod, configurations are scripted in Starlark, a Python dialect by Google also used by Bazel and Buck build systems. Isopod offers runtime built-ins to access services and utilities such as Vault secret management, Kubernetes apiserver, HTTP requester, Base64 encoder, and UUID generator, etc. Isopod uses separate runtime for unit tests to mock all built-ins, providing the test coverage not possible before.

A 5-min read, this medium post explains the inefficiency of existing YAML templating tools when dealing with values not statically known and complicated control logics such as loops and branches. It also gives simple code examples to show why Isopod is an expressive, hermetic, and extensible solution to configuration management in Kubernetes.

• Isopod
• Build
• Main Entryfile
• Clusters
  • gke()
  • onprem()
• Addons
• Generate Addons
• Load Remote Isopod Modules
• Built-ins
• kube
  • Methods:
  • kube.put
  • kube.delete
  • kube.put_yaml
  • kube.get
  • kube.exists
  • kube.from_str, kube.from_int
• Vault
  • Methods:
  • vault.read
  • vault.write
  • vault.exist
• Helm
  • Methods:
  • helm.apply
• Misc
  • base64.{encode, decode}
  • uuid.{v3, v4, v5}
  • http.{get, post, patch, put, delete}
  • hash.{sha256, sha1, md5}
  • sleep
  • error
• Testing
• Dry Run Produces YAML Diffs
• Diff filtering
• License
• Contributions

Build

$ go version
go version go1.14 darwin/amd64
$ GO111MODULE=on go build

Main Entryfile

Isopod will call the clusters(ctx) function in the main Starlark file to get a list of target clusters. For each of such clusters, isopod will call addons(ctx) to get a list of addons for configuration rollout.

Example:

CLUSTERS = [
    onprem(env="dev", cluster="minikube", vaultkubeconfig="secret/path"),
    gke(
        env="prod",
        cluster="paas-prod",
        location="us-west1",
        project="cruise-paas-prod",
        use_internal_ip="false", # default to "false", which uses public endpoint
    ),
]

def clusters(ctx):
    if ctx.cluster != None:
        return [c for c in CLUSTERS if c.cluster == ctx.cluster]
    elif ctx.env != None:
        return [c for c in CLUSTERS if c.env == ctx.env]
    return CLUSTERS

def addons(ctx)
    return [
        addon("ingress", "configs/ingress.ipd", ctx),
    ]

Clusters

The ctx argument to clusters(ctx) comes from the command line flag --context to Isopod. This flag takes a comma-separated list of foo=bar and makes these values available in Starlark as ctx.foo (which gives "bar"). Currently Isopod supports the following clusters, and could easily be extended to cover other Kubernetes vendors, such as EKS and AKS.

gke()

Represents a Google Kubernetes Engine. Authenticates using Google Cloud Service Account Credentials or Google Default Application Credentials. Requires the cluster, location and project fields, while optionally takes use_internal_ip field to connect API server via private endpoint. Additional fields are allowed.

onprem()

Represents an on-premise or self-managed Kubernetes cluster. Authenticates using the kubeconfig file or Vault path containing the kubeconfig. No fields are required, though setting the vaultkubeconfig field to the path in Vault where the KubeConfig exists is necessary to utilize this auth method.

Addons

The ctx argument to addons(ctx) contains all fields of the chosen cluster. For example, say the cluster is

gke(
    env="prod",
    cluster="paas-prod",
    location="us-west1",
    project="cruise-paas-prod",
    use_internal_ip="false", # default to "false", which uses public endpoint
),

Then, each addon may access the cluster information as ctx.env to get "prod" and ctx.location to get "us-west1". Accessing nonexistant attribute ctx.foo will get None.

Each addon is represented using the addon() Starlark built-in, which takes three arguments, for example addon("name", "entry_file.ipd", ctx). The first argument is the addon name, used by the --match_addon feature. The thrid is optional and represents the ctx input to addons(ctx) to make the cluster attributes available to the addon. Each addon must implement install(ctx) and remove(ctx) functions.

More advanced examples can be found in the examples folder.

Example Nginx addon:

appsv1 = proto.package("k8s.io.api.apps.v1")
corev1 = proto.package("k8s.io.api.core.v1")
metav1 = proto.package("k8s.io.apimachinery.pkg.apis.meta.v1")

def install(ctx):
    metadata = metav1.ObjectMeta(
        name="nginx",
        namespace="example",
        labels={"app": "nginx"},
    )

    nginxContainer = corev1.Container(
        name=metadata.name,
        image="nginx:1.15.5",
        ports=[corev1.ContainerPort(containerPort=80)],
    ),

    deploySpec = appsv1.DeploymentSpec(
        replicas=3,
        selector=metav1.LabelSelector(matchLabels=metadata.labels),
        template=corev1.PodTemplateSpec(
            metadata=metadata,
            spec=corev1.PodSpec(
                containers=[nginxContainer],
            ),
        ),
    )

    kube.put(
        name=metadata.name,
        namespace=metadata.namespace,
        data=[appsv1.Deployment(
            metadata=metav1.ObjectMeta(name=metadata.name),
            spec=deploySpec,
        )],
    )

Generate Addons

You might come from a place where you have a yaml file, but you want to derive an isopod addon from it. It can be cumbersome to re-write huge yaml files in Starlark. So isopod offers a convenience command to generate the Starlark code based on a yaml or json input file containing any kubernetes API object:

isopod generate runtime/testdata/clusterrolebinding.yaml > addon.ipd

For now all k8s.io resources are supported.

Load Remote Isopod Modules

Similar to Bazel's WORKSPACE file, the isopod.deps file allows you to define remote and versioned git modules to import to local modules. For example,

git_repository(
    name="isopod_tools",
    commit="dbe211be57bc27b947ab3e64568ecc94c23a9439",
    remote="https://github.com/cruise-automation/isopod.git",
)

To import remote modules, use load("@target_name//path/to/file", "foo", "bar"), for example,

load("@isopod_tools//examples/helpers.ipd",
     "health_probe", "env_from_field", "container_port")

...
spec=corev1.PodSpec(
    containers=[corev1.Container(
        name="nginx-ingress-controller",
        image="quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0",
        env=[
            env_from_field("POD_NAME", "metadata.name"),
            env_from_field("POD_NAMESPACE", "metadata.namespace"),
        ],
        livenessProbe=health_probe(10254),
        readinessProbe=health_probe(10254),
        ports=[
            container_port("http", 80),
            container_port("https", 443),
            container_port("metrics", 10254),
        ],
    )],
)

To import remote addon files, use addon("addon_name", ""@addon_name//path/to/file", ctx), for example,

isopod.deps:

git_repository(
    name="versioned_addon",
    commit="1.0.0",
    remote="https://github.com/cruise-automation/addon.git",
)
...

main.ipd:

def addons(ctx):
    if ctx.cluster == None:
        error("`ctx.cluster' not set")
    if ctx.foobar != None:
        error("`ctx.foobar' must be `None', got: {foobar}".format(
            foobar=ctx.foobar))

    return [
        addon("addon_name", "@addon_name//addon/addon.ipd", ctx),
    ]

By default Isopod uses $(pwd)/isopod.deps, which you can override with --deps flag.

Built-ins

Built-ins are pre-declared packages available in Isopod runtime. Typically they perform I/O to Kubernetes, Vault, GCP and other resources but could be used for break-outs into other operations not supported by the main Starlark interpreter.

Currently these build-ins are supported:

kube

Built-in for managing Kubernetes objects.

Methods:

kube.put

Updates (creates if it doesn't already exist) object in Kubernetes.

kube.put(
    name = "nginx-role",
    namespace = "nginx-ingress",
    # Optional Kubernetes API Group parameter. If not set, will attempt to
    # deduce the group from message type but since Kubernetes API Group names
    # are highly irregular, this may fail.
    api_group = 'rbac.authorization.kubernetes.io',
    data = [
        rbacv1.Role(),
    ],
)

Supported args: + name - Name (.metadata.name) of the resource + namespace (Optional) - Namespace (.metadata.namespace) of the resource + api_group (Optional) - API group of the resource. If not provided, Isopod runtime will attempt to deduce the resource from just Proto type name which is unreliable. It is recommended to set this for all objects outside of core group. Optionally, version can also be specified after a /, example: + apiextensions.k8s.io - specify the group only, version is implied from Proto or from runtime. + apiextensions.k8s.io/v1 - specify both group and version. + subresource (Optional) - A subresource specifier (e.g /status). + data - A list of Protobuf definitions of objects to be created.

kube.delete

Deletes object in Kubernetes.

# kwarg key is resource name, value is <namespace>/<name> (just <name> for
# non-namespaced resources).
kube.delete(deployment="default/nginx")
# api_group can optionally be provided to remove ambuguity (if multiple
# resources by the same name exist in different API Groups or different versions).
kube.delete(clusterrole="nginx", api_group = "rbac.authorization.k8s.io/v1")

kube.put_yaml

Same as put but for YAML/JSON data. To be used for CRDs and other custom types. kube.put usage is preferred for the standard set of Kubernetes types.

ark_config = """
apiVersion: ark.heptio.com/v1
kind: Config
metadata:"
  namespace: ark-backup
  name: default
backupStorageProvider:
  name: gcp
  bucket: test-ark-backup
persistentVolumeProvider:
  name: gcp
"""

kube.put_yaml(
    name = "ark-config",
    namespace = "backup",
    data = [ark_config])

# Alternatively render from native Starlark struct object via JSON:
ark_config = struct(
    apiVersion = "ark.heptio.com/v1",
    kind = "Config",
    metadata = struct(
        name = "ark-backup",
        namespace = "default",
    ),
    backupStorageProvider = struct(
        name = "gcp",
        bucket = "test-ark-backup",
    ),
    persistentVolumeProvider = struct(
        name = "gcp",
    ),
)

kube.put_yaml(
    name = "ark-config",
    namespace = "backup",
    data = [ark_config.to_json()])

kube.get

Reads object from API Server. If wait argument is set to duration (e.g 10s) will block until the object is successfully read or timer expires. If json=True optional argument is provided, will render object as unstructured JSON represented as Starlark dict at top level. This is useful for CRDs as they typically do not support Protobuf representation.

# Wait 60s for Service Account token secret.
secret = kube.get(secret=namespace+"/"+serviceaccount.secrets[0].name, wait="60s")

# Get ClusterRbacSyncConfig CRD.
cadmin = kube.get(clusterrbacsyncconfig="cluster-admin",
                  api_group="rbacsync.getcruise.com",
                  json=True)

It is also possible to receive a list of kubernetes objects. They can be filtered as defined in the API documentation.

# Get all pods in namespace kube-system.
pods = kube.get(pod="kube-system/")

# Get all pods with label component=kube-apiserver
pods = kube.get(pod="kube-system/?labelSelector=component=kube-apiserver")

kube.exists

Checks whether a resource exists. If wait argument is set to duration (e.g 10s) will block until the object is successfully read or timer expires.

```python

Assert that the resource doesn't exist.

e = kube.exists(secret=namespace+"/"+serviceaccount.secrets[0].name, wait