amazon-vpc-cni-k8s

Networking plugin for pod networking in Kubernetes using Elastic Network Interfaces on AWS.

Setup

Download the latest version of the yaml and apply it to the cluster.

kubectl apply -f aws-k8s-cni.yaml

Launch kubelet with network plugins set to cni (--network-plugin=cni), the cni directories configured (--cni-config-dir and --cni-bin-dir) and node ip set to the primary IPv4 address of the primary ENI for the instance (--node-ip=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)). It is also recommended that you set --max-pods equal to (the number of ENIs for the instance type × (the number of IPs per ENI - 1)) + 2; for details, see vpc_ip_resource_limit.go. Setting --max-pods will prevent scheduling that exceeds the IP address resources available to the kubelet.

The default manifest expects --cni-conf-dir=/etc/cni/net.d and --cni-bin-dir=/opt/cni/bin.

Alternatively there is also a Helm chart: eks/aws-vpc-cni

IAM Policy

See here for required IAM policies.

Building

• make defaults to make build-linux that builds the Linux binaries.
• unit-test, format,lint and vet provide ways to run the respective tests/tools and should be run before submitting a PR.
• make docker will create a docker container using docker buildx that contains the finished binaries, with a tag of amazon/amazon-k8s-cni:latest
• make docker-unit-tests uses a docker container to run all unit tests.
• Builds for all build and test actions run in docker containers based on .go-version unless a different GOLANG_IMAGE tag is passed in.

Components

There are 2 components:

• CNI Plugin, which will wire up the host's and pod's network stack when called.
• ipamd, a long-running node-Local IP Address Management (IPAM) daemon, is responsible for:
• maintaining a warm-pool of available IP addresses, and
• assigning an IP address to a Pod.

The details can be found in Proposal: CNI plugin for Kubernetes networking over AWS VPC.

Help & Feedback

For help, please consider the following venues (in order):

• Amazon EKS Best Practices Guide for Networking
• Troubleshooting Guide provides tips on how to debug and troubleshoot this CNI.
• Search open issues
• File an issue
• Chat with us on the #aws-vpc-cni channel in the Kubernetes Slack community.

Recommended Version

For all Kubernetes releases, we recommend installing the latest VPC CNI release. The following table denotes our oldest recommended VPC CNI version for each actively supported Kubernetes release.

Version Upgrade

Upgrading (or downgrading) the VPC CNI version should result in no downtime. Existing pods should not be affected and will not lose network connectivity. New pods will be in pending state until the VPC CNI is fully initialized and can assign pod IP addresses. In v1.12.0+, VPC CNI state is restored via an on-disk file: /var/run/aws-node/ipam.json. In lower versions, state is restored via calls to container runtime.

ENI Allocation

When a worker node first joins the cluster, there is only 1 ENI along with all of the addresses on the ENI. Without any configuration, ipamd always tries to keep one extra ENI.

When the number of pods running on the node exceeds the number of addresses on a single ENI, the CNI backend starts allocating a new ENI using the following allocation scheme:

• If the number of current running Pods is between 0 and 29, ipamd will allocate one more eni. And Warm-Pool size is 2 eni * (30 -1) = 58
• If the number of current running Pods is between 30 and 58, ipamd will allocate 2 more eni. And Warm-Pool size is 3 eni * (30 -1) = 87

For example, a m4.4xlarge node can have up to 8 ENIs, and each ENI can have up to 30 IP addresses. See Elastic Network Interfaces documentation for details.

For a detailed explanation, see WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET.

Privileged mode

VPC CNI makes use of privileged mode (privileged: true) in the manifest for its aws-vpc-cni-init and aws-eks-nodeagent containers. aws-vpc-cni-init container requires elevated privilege to set the networking kernel parameters while aws-eks-nodeagent container requires these privileges for attaching BPF probes to enforce network policy

Network Policies

In Kubernetes, by default, all pod-to-pod communication is allowed. Communication can be restricted with Kubernetes NetworkPolicy objects.

VPC CNI versions v1.14, and greater support Kubernetes Network Policies.. Network Policies specify how pods can communicate over the network, at the IP address or port level. The VPC CNI implements the Kubernetes NetworkPolicy API. Network policies generally include a pod selector, and Ingress/Egress rules.

For EKS clusters, review Configure your cluster for Kubernetes network policies in the Amazon EKS User Guide.

The AWS VPC CNI implementation of network policies may be enabled in self-managed clusters. This requires the VPC CNI agent, the Network Policy Controller, and Network Policy Node Agent.

Review the Network Policy FAQ for more information.

Network Policy Related Components

• Network Policy Controller watches for NetworkPolicy objects and instructs the node agent.
• Network policy controller configures policies for pods in parallel to pod provisioning, until then new pods will come up with default allow policy. All ingress and egress traffic is allowed to and from the new pods until they are resolved against the existing policies.
• This controller is automatically installed on the EKS Control Plane.
• Network Policy Node Agent implements Network Policies on nodes by creating eBPF programs.
• AWS eBPF SDK for Go provides an interface to interact with eBPF programs on the node. This SDK allows for runtime introspection, tracing, and analysis of eBPF execution, aiding in identifying and resolving connectivity issues.
• VPC Resource Controller manages Branch & Trunk Network Interfaces for Kubernetes Pods.

ConfigMap

If the VPC CNI is installed as an Amazon EKS add-ons (also known as a managed add-on), configure it using AWS APIs as described in the EKS User Guide.

If the VPC CNI is installed with a Helm Chart, the ConfigMap is installed in your cluster. Review the Helm Chart information.

Otherwise, the VPC CNI may be configured with a ConfigMap, as shown below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  enable-network-policy-controller: "true"

• enable-network-policy-controller
• Default: False
• If enabled, the VPC CNI will enforce NetworkPolicies. Pods that are selected by at least one NetworkPolicy will have their traffic restricted.

Helm Charts

AWS publishes a Helm chart to install the VPC CNI. Review how to install the helm chart, and the configuration parameters for the chart.

CNI Configuration Variables

The Amazon VPC CNI plugin for Kubernetes supports a number of configuration options, which are set through environment variables. The following environment variables are available, and all of them are optional.

AWS_MANAGE_ENIS_NON_SCHEDULABLE (v1.12.6+)

Type: Boolean as a String

Default: false

Specifies whether IPAMD should allocate or deallocate ENIs on a non-schedulable node.

AWS_VPC_CNI_NODE_PORT_SUPPORT

Type: Boolean as a String

Default: true

Specifies whether NodePort services are enabled on a worker node's primary network interface. This requires additional iptables rules, and the kernel's reverse path filter on the primary interface is set to loose.

AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG

Type: Boolean as a String

Default: false

Specifies that your pods may use subnets and security groups that are independent of your worker node's VPC configuration. By default, pods share the same subnet and security groups as the worker node's primary interface. Setting this variable to true causes ipamd to use the security groups and VPC subnet in a worker node's ENIConfig for elastic network interface allocation. You must create an ENIConfig custom resource for each subnet that your pods will reside in, and then annotate or label each worker node to use a specific ENIConfig. Multiple worker nodes can be annotated or labelled with the same ENIConfig, but each Worker node can be annotated with a single ENIConfig at a time. Further, the subnet in the ENIConfig must belong to the same Availability Zone that the worker node resides in. For more information, see CNI Custom Networking in the Amazon EKS User Guide.

ENI_CONFIG_ANNOTATION_DEF

Type: String

Default: k8s.amazonaws.com/eniConfig

Specifies node annotation key name. This should be used when AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true. Annotation value will be used to set ENIConfig name. Note that annotations take precedence over labels.

ENI_CONFIG_LABEL_DEF

Type: String

Default: k8s.amazonaws.com/eniConfig

Specifies node label key name. This should be used when AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true. Label value will be used to set ENIConfig name. Note that annotations will take precedence over labels. To use labels, ensure there is no annotation with key k8s.amazonaws.com/eniConfig or defined key (in ENI_CONFIG_ANNOTATION_DEF) set on the node. To select an ENIConfig based upon availability zone set this to topology.kubernetes.io/zone and create an ENIConfig custom resource for each availability zone (e.g. us-east-1a). Note that tag failure-domain.beta.kubernetes.io/zone is deprecated and replaced with the tag topology.kubernetes.io/zone.

HOST_CNI_BIN_PATH

Type: String

Default: /host/opt/cni/bin

Specifies the location to install CNI binaries. Note that the aws-node daemonset mounts /opt/cni/bin to /host/opt/cni/bin. The value you choose must be a location that the aws-node pod can write to.

HOST_CNI_CONFDIR_PATH

Type: String

Default: /host/etc/cni/net.d

Specifies the location to install the VPC CNI conflist. Note that the aws-node daemonset mounts /etc/cni/net.d to /host/etc/cni/net.d. The value you choose must be a location that the aws-node pod can write to.

AWS_VPC_ENI_MTU (v1.6.0+)

Type: Integer as a String

Default: 9001

Used to configure the MTU size for attached ENIs. The valid range for IPv4 is from 576 to 9001, while the valid range for IPv6 is from 1280 to 9001.

AWS_VPC_K8S_CNI_EXTERNALSNAT

Type: Boolean as a String

Default: false

Specifies whether an external NAT gateway should be used to provide SNAT of secondary ENI IP addresses. If set to true, the SNAT iptables rule and off-VPC IP rule are not applied, and these rules are removed if they have already been applied. SNAT can be disabled in scenarios where pods need direct access to external networks (such as VPN, Direct Connect, or other VPCs) without NAT translation, and where pods are not expected to require direct Internet access via an Internet Gateway. When SNAT is disabled, nodes are typically placed in private subnets, with outbound Internet connectivity provided through an AWS NAT Gateway or another external NAT device.

AWS_VPC_K8S_CNI_RANDOMIZESNAT

Type: String

Default: prng

Valid Values: hashrandom, prng, none

Specifies whether the SNAT iptables rule should randomize the outgoing ports f