AWS Secrets Manager and Config Provider for Secret Store CSI Driver

AWS offers two services to manage secrets and parameters conveniently in your code. AWS Secrets Manager allows you to easily rotate, manage, and retrieve database credentials, API keys, certificates, and other secrets throughout their lifecycle. AWS Systems Manager Parameter Store provides hierarchical storage for configuration data. The AWS provider for the Secrets Store CSI Driver allows you to make secrets stored in Secrets Manager and parameters stored in Parameter Store appear as files mounted in Kubernetes pods.

Installation

Requirements

• Amazon Elastic Kubernetes Service (EKS) 1.17+ running an EC2 node group (Fargate node groups are not supported [^1]). If using EKS Pod Identity feature, EKS 1.24+ is required.
• Secrets Store CSI driver installed: shell helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver Note that older versions of the driver may require the --set grpcSupportedProviders="aws" flag on the install step.
  Note this step can be skipped if installing from Helm. The Helm chart for the ASCP by default automatically installs a compatible version of the Secrets Store CSI driver as a helm dependency. This can be disabled by setting secrets-store-csi-driver.install=false.
• IAM Roles for Service Accounts (IRSA) or EKS Pod Identity as described in the usage section below.

[^1]: The CSI Secret Store driver runs as a DaemonSet, and as described in the AWS documentation, DaemonSet is not supported on Fargate.

Installing the AWS Provider and Config Provider (ASCP)

Option 1: Using helm

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws

Option 2: Using kubectl

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Separate CSI Driver Installation

If you install the secrets-store-csi-driver separately (not via this Helm chart), you must configure tokenRequests in the CSI driver for the AWS provider to authenticate with AWS services:

helm upgrade csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --set tokenRequests[0].audience="sts.amazonaws.com" \
  --set tokenRequests[1].audience="pods.eks.amazonaws.com"

Or if using kubectl, add the following to your CSIDriver manifest:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: secrets-store.csi.k8s.io
spec:
  tokenRequests:
    - audience: "sts.amazonaws.com"
    - audience: "pods.eks.amazonaws.com"

Usage

Set the region name and name of your cluster to use in the bash commands that follow:

REGION=<REGION>
CLUSTERNAME=<CLUSTERNAME>

Where <REGION> is the region in which your Kubernetes cluster is running and <CLUSTERNAME> is the name of your cluster.

Create a test secret:

aws --region "$REGION" secretsmanager  create-secret --name MySecret --secret-string '{"username":"memeuser", "password":"hunter2"}'

Create an access policy for the pod scoped down to just the secrets it should have and save the policy ARN in a shell variable:

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{
    "Version": "2012-10-17",
    "Statement": [ {
        "Effect": "Allow",
        "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
        "Resource": ["arn:*:secretsmanager:*:*:secret:MySecret-??????"]
    } ]
}')

Note, when using SSM parameters the permission "ssm:GetParameters" is needed in the policy. To simplify this example we use wild card matches above but you could lock this down further using the full ARN from the output of create-secret above.

Option 1: Using IAM Roles For Service Accounts (IRSA)

1. Create the IAM OIDC provider for the cluster if you have not already done so:

eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once

1. Next, create the service account to be used by the pod and associate the above IAM policy with that service account. For this example we use nginx-irsa-deployment-sa for the service account name:

eksctl create iamserviceaccount --name nginx-irsa-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

For a private cluster, ensure that the VPC the cluster is in has an AWS STS endpoint. For more information, see Interface VPC endpoints in the AWS IAM User Guide.

1. Create the SecretProviderClass which tells the AWS provider which secrets are to be mounted in the pod. The ExampleSecretProviderClass-IRSA.yaml in the examples directory will mount "MySecret" created above:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass-IRSA.yaml

1. Finally, we can deploy our pod. The ExampleDeployment-IRSA.yaml in the examples directory contains a sample nginx deployment that mounts the secrets under /mnt/secrets-store in the pod:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment-IRSA.yaml

To verify the secret has been mounted properly, See the example below:

kubectl exec -it $(kubectl get pods | awk '/nginx-irsa-deployment/{print $1}' | head -1) -- cat /mnt/secrets-store/MySecret; echo

Option 2: Using EKS Pod Identity

Note: EKS Pod Identity option is only supported for EKS in the Cloud. It's not supported for Amazon EKS Anywhere, Red Hat Openshift Service on AWS (ROSA) and self-managed Kubernetes clusters on Amazon Elastic Compute Cloud (Amazon EC2) instances. 1. Install Amazon EKS Pod Identity Agent Add-on on the cluster.

eksctl create addon --name eks-pod-identity-agent --cluster "$CLUSTERNAME" --region "$REGION"

1. Create an IAM role that can be assumed by the Amazon EKS service principal for Pod Identity and attach the above IAM policy to grant access to the test secret.

ROLE_ARN=$(aws --region "$REGION" --query Role.Arn --output text iam create-role --role-name nginx-deployment-role --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}')

aws iam attach-role-policy \
    --role-name nginx-deployment-role \
    --policy-arn $POLICY_ARN

1. Next create the service account to be used by the pod and associate the service account with the IAM role created above. For this example we use nginx-pod-identity-deployment-sa for the service account name:

eksctl create podidentityassociation \
    --cluster "$CLUSTERNAME" \
    --namespace default \
    --region "$REGION" \
    --service-account-name nginx-pod-identity-deployment-sa \
    --role-arn $ROLE_ARN \
    --create-service-account true

1. Now create the SecretProviderClass which tells the AWS provider which secrets are to be mounted in the pod. The ExampleSecretProviderClass-PodIdentity.yaml in the examples directory will mount "MySecret" created above:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass-PodIdentity.yaml

1. Finally, we can deploy our pod. The ExampleDeployment-PodIdentity.yaml in the examples directory contains a sample nginx deployment that mounts the secrets under /mnt/secrets-store in the pod:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment-PodIdentity.yaml

To verify the secret has been mounted properly, See the example below:

kubectl exec -it $(kubectl get pods | awk '/nginx-pod-identity-deployment/{print $1}' | head -1) -- cat /mnt/secrets-store/MySecret; echo

Troubleshooting

Most errors can be viewed by describing the pod deployment. For the deployment, find the pod names using get pods (use -n <NAMESPACE> if you are not using the default namespace):

kubectl get pods

Then describe the pod (substitute the pod ID from above for <PODID>, as before use -n if you are not using the default namespace):

kubectl describe pod/<PODID>

Additional information may be available in the provider logs:

kubectl -n kube-system get pods
kubectl -n kube-system logs pod/<PODID>

Where <PODID> in this case is the id of the csi-secrets-store-provider-aws pod.

SecretProviderClass options

The SecretProviderClass has the following format:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: <NAME>
spec:
  provider: aws
  parameters:

The parameters section contains the details of the mount request and contain one of the three fields: * objects: This is a string containing a YAML declaration (described below) of the secrets to be mounted. This is most easily written using a YAML multi-line string or pipe character. For example: yaml parameters: objects: | - objectName: "MySecret" objectType: "secretsmanager" * region: An optional field to specify the AWS region to use when retrieving secrets from Secrets Manager or Parameter Store. If this field is missing, the provider will lookup the region from the topology.kubernetes.io/region label on the node. This lookup adds overhead to mount requests so clusters using large numbers of pods will benefit from providing the region here. * failoverRegion: An optional field to specify a secondary AWS region to use when retrieving secrets. See the Automated Failover Regions section in this readme for more information. * pathTranslation: An optional field to specify a substitution character to use when the path separator character (slash on Linux) is used in the file name. If a Secret or parameter name contains the path separator failures will occur when the provider tries to create a mounted file using the name. When not specified the underscore character is used, thus My/Path/Secret will be mounted as My_Path_Secret. This pathTranslation value can either be the string "False" or a single character string. When set to "False", no character substitution is performed. * usePodIdentity: An optional field that determines the authentication approach. When not specified, it defaults to using IAM Roles for Service Accounts (IRSA). - To use EKS Pod Identity, use any of these values: "true", "True", "TRUE", "t", "T". - To explicitly use IRSA, set to any of these values: "false", "False", "FALSE", "f", or "F". * preferredAddressType: An optional field that specifies the preferred IP address type for Pod Identity Agent endpoint communication. The field is only applicable when using EKS Pod Identity feature and will be ignored when using IAM Roles for Service Accounts.Values are case-insensitive. Valid values are: - "ipv4", "IPv4", or "IPV4" - Force the use of Pod Identity Agent IPv4 endpoint - "ipv6", "IPv6", or "IPV6" - Force the use of Pod Identity Agent IPv6 endpoint - not specified - Use auto endpoint selection, trying IPv4 endpoint first and falling back to IPv6 endpoint if IPv4 fails

The primary objects field of the SecretProviderClass can contain the following sub-fields: * objectName: This field is required. It specifies the name of the secret or parameter to be fetched. For Secrets Manager this is the SecretId parameter and can be either the friendly name or full ARN of the secret. For SSM Parameter Store, this is the Name of the parameter and can be either the name or full ARN of the parameter. * objectType: This field is optional when using a Secrets Manager ARN for objectName, otherwise it is required. This field can be either "secretsmanager" or "ssmparameter". * objectAlias: This optional field specifies the file name under which the secret will be mounted. When not specified the file name defaults to objectName. * filePermission: This optional fie