sample-external-issuer

External issuers extend cert-manager to issue certificates using APIs and services which aren't built into the cert-manager core.

This repository provides an example of an [External Issuer].

Install

kubectl apply -f https://github.com/cert-manager/sample-external-issuer/releases/download/v0.1.0/install.yaml

Demo

You can run the sample-external-issuer on a local cluster with this command:

make kind-cluster deploy-cert-manager docker-build kind-load deploy e2e

How to write your own external issuer

If you are writing an external issuer you may find it helpful to review the code and the commits in this repository and to follow the steps below, replacing references to sample-external-issuer with the name of your project.

Prerequisites

You will need the following command line tools installed on your PATH:

• Git
• Golang v1.20+
• Docker v17.03+
• Kind v0.18.0+
• Kubectl v1.26.3+
• Kubebuilder v3.9.1+
• Kustomize v3.8.1+

You may also want to read: the [Kubebuilder Book] and the [cert-manager Concepts Documentation] for further background information.

Create a test cluster

We will need a Kubernetes cluster on which to test our issuer and we can quickly create one using kind.

kind create cluster

This will update your KUBECONFIG file with the URL and credentials for the test cluster. You can explore it using kubectl

kubectl get nodes

This should show you details of a single node.

Create a repository

We need a Git repository to track changes to the issuer code. You can start by creating a repository on GitHub or you can create it locally.

mkdir sample-external-issuer
cd sample-external-issuer
git init

Initialise a Go mod file

A Go project needs a go.mod file which defines the root name of your Go packages.

go mod init github.com/cert-manager/sample-external-issuer

Initialise a Kubebuilder project

kubebuilder init  --domain example.com --owner 'The cert-manager Authors'

This will create multiple directories and files for building and deploying your project. Notably, these files include:

• Makefile: various commands useful for development and deployment
• config/: various kustomize configuration files.
• Dockerfile: used to statically compile the issuer and package it as a "distroless" Docker image.
• main.go: the issuer's main entry point.

Run the controller-manager

With all these tools in place and with the project initialised you should now be able to run the issuer for the first time.

make run

This will compile and run the issuer locally and it will connect to the test cluster and log some startup messages. We will add more to it in the next steps.

Creating Issuer and ClusterIssuer custom resources

An [External Issuer] must implement two custom resources for compatibility with cert-manager: Issuer and ClusterIssuer

NOTE: It is important to understand the [Concept of Issuers] before proceeding.

We create the custom resource definitions (CRDs) using kubebuilder using the following commands; you'll be prompted to create APIs and controllers and you can answer y to all of the prompts.

kubebuilder create api --group sample-issuer --kind Issuer --version v1alpha1
kubebuilder create api --group sample-issuer --kind ClusterIssuer --version v1alpha1 --namespaced=false

The values we pass to these commands specify the GVK (group, version, kind):

• group is the name given to a collection of custom resource APIs
• kind is the name of an individual resource in that group
• version allows you to create multiple versions of your APIs as they evolve, whilst providing backwards compatibility for clients using older API versions

These commands will have created some boilerplate files and directories: api/ and controllers/, which we now need to edit as follows:

• api/v1alpha1/clusterissuer_types.go: Remove the ClusterIssuerSpec and ClusterIssuerStatus and replace them with IssuerSpec and IssuerStatus. This is because both types of issuers share the same configuration and status reporting.

• controllers/{cluster}issuer_controller.go: Edit the Kubebuilder RBAC Markers. The controller should not have write permission to the Issuer or ClusterIssuer. It should only be permitted to modify the Status subresource.

• api/v1alpha1/{cluster}issuer_types.go: And finally, remove any placeholder comments from the API files.

After modifying [Kubebuilder Markers] and API source files you should always regenerate all generated code and configuration, as follows:

make generate manifests

You should see a number of new and modified files, reflecting the changes you made to the API source files and to the markers.

Creating a CertificateRequest controller

We now need a controller to handle cert-manager CertificateRequest resources. This controller will watch for CertificateRequest resources and attempt to sign their attached certificate signing requests (CSR).

Your external issuer will likely interact with your certificate authority using a REST API, so this is the controller where we will eventually need to instantiate an HTTP client, directly or via an API wrapper library. And we will need to get the configuration and credentials for this from the Issuer or ClusterIssuer referred to by the CertificateRequest.

Start by copying the controllers/issuer_controller.go to controllers/certificaterequest_controller.go and modifying its code and comments to refer to CertificateRequest rather than Issuer.

NOTE: You will need to import the cert-manager V1 API and this in turn will pull in a number of transitive dependencies of cert-manager which we will deal with shortly.

import (
...
    cmapi "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1"
...
)

Next edit main.go and register the new CertificateRequestReconciler in the same way that the IssuerReconciler is registered. You will also need to add the cert-manager API types to the Scheme:

func init() {
...
    utilruntime.Must(cmapi.AddToScheme(scheme))
...
}

The Scheme is how the controller-runtime client knows how to decode and encode the API resources from the Kubernetes API server. So it is important to add all the API types that are used in your issuer.

Finally run make generate manifests again to update all the generated code.

Get the CertificateRequest

The CertificateRequestReconciler is triggered by changes to any CertificateRequest resource in the cluster. The Reconcile function is called with the name of the object that changed, and the first thing we need to do is to GET the complete object from the Kubernetes API server.

The Reconcile function may occasionally be triggered with the names of deleted resources, so we have to handle that case gracefully.

Explore the unit-tests in controllers/certificaterequest_test.go and try running the tests and seeing them fail before updating the controller code. These are table-driven tests which will execute the Reconcile function many times, with inputs supplied as function arguments and also certain inputs that come from the parent object. It also uses a FakeClient which can be primed with a collection of Kubernetes API objects. The test will check the output of the Reconcile function for errors and later we will make it check the changes that have been made to the supplied API objects.

In the implementation we are careful to return Result{}, nil when the CertificateRequest is not found. This tells controller-runtime do not retry. Other error types are assumed to be temporary errors and are returned.

NOTE: If you return an error, controller-runtime will retry with an increasing backoff, so it is very important to distinguish between temporary and permanent errors.

Ignore foreign CertificateRequest

We only want to reconcile CertificateRequest resources that are configured for our external issuer. So the next piece of controller logic attempts to exit early if CertificateRequest.Spec.IssuerRef does not refer to our particular Issuer or ClusterIssuer types.

As before explore the unit-tests and see how we modify the success case, where the IssuerRef does refer to one of our types. And then we add some error cases where the Group or the Kind are unrecognised.

Also note how in the implementation we use the Scheme.New method to verify the Kind. This later will allow us to easily handle both Issuer and ClusterIssuer references.

If there is a mismatch in the IssuerRef we ignore the CertificateRequest.

Check that the CertificateRequest is Approved

Issuers must only sign Approved CertificateRequest resources. If the CertificateRequest has been Denied, then the Issuer should set a Ready condition to False, and set the FailureTime. If the CertificateRequest has been Approved, then the Issuer should process the request.

Issuers are not responsible for approving CertificateRequests. You can read more about the CertificateRequest Approval API in the cert-manager documentation.

Adjust your CertificateRequest controller and the accompanying unit tests to ignore CertificateRequest resources if they are Denied or if they are not Approved. Use the cert-manager API utility package which contains functions for checking the Approved and Denied conditions of a CertificateRequest.

Set the CertificateRequest Ready condition

The [External Issuer] documentation says the following:

It is important to update the condition status of the CertificateRequest to a ready state, as this is what is used to signal to higher order controllers, such as the Certificate controller, that the resource is ready to be consumed. Conversely, if the CertificateRequest fails, it is important to mark the resource as such, as this will also be used to signal to higher order controllers.

So now we need to ensure that our issuer always sets one of the strongly defined conditions on all the CertificateRequest referring to our Group.

Study the changes and the additional tests. Note that the first thing we do is check whether the Ready condition is already true in which case we can exit early. Note also the use of a defer function which ensures that the condition is always set and that it is always set to false if an error has occurred.

Get the Issuer or ClusterIssuer

The Issuer or ClusterIssuer for the CertificateRequest will usually contain configuration that you will need to connect to your certificate authority API. It may also contain a reference to a Secret containing credentials which you will use to authenticate with with your certificate authority API.

So now we attempt to GET the Issuer or ClusterIssuer and to do this we need to derive a resource name. An Issuer has both a name and a namespace. A ClusterIssuer is cluster scoped and does not have a namespace. So we check which type we have in order to derive the correct name.

If the GET request fails, we return the error so as to trigger the retry-with-backoff behaviour (described above). This allows for situations where the CertificateRequest may have been created before the corresponding Issuer. This case is demonstrated in the unit tests.

Check the Issuer or ClusterIssuer Ready condition

An issuer will often perform some initialisation when it is first created, for example it might create a private key and CA certificate and store those somewhere, and such operations take time. So we give the Issuer and ClusterIssuer resources their own Ready conditions which the IssuerReconciler can set to signal that the initialization is complete and that the issuer is ready and healthy.

The CertificateRequestReconciler should then wait for the Issuer to be Ready before progressing further.

Get the Issuer or ClusterIssuer credentials from a Secret

The API for your CA may require some configuration and credentials and the obvious place to store these is in a Kubernetes Secret. We extend the IssuerSpec to include a URL field and a AuthSecretName, which is the name of a Secret. As usual run make generate manifests after modifying the API source files:

make generate manifests

NOTE: The namespace of that Secret is deliberately not specified here, because that would breach a security boundary and potentially allow someone who has permission to create Issuer resources, to make the controller access secrets in another namespace which that person would not normally have access to.

For this reason, the Secret for an Issuer MUST be in the same namespace as the Issuer. T