k3sup 🚀 (said 'ketchup')

k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

The tool is written in Go and is cross-compiled for Linux, Windows, MacOS and even on Raspberry Pi.

How do you say it? Ketchup, as in tomato.

Introducing K3sup Pro 🎉

Whilst the CE edition is ideal for experimentation, k3sup pro was built to satisfy long standing requests for an IaaC/GitOps experience.

k3sup pro adds a plan and apply command to automate installations both small and large - running in parallel. The plan file can be customised and retained in Git for maintenance and updates.

Contents:

• k3sup 🚀 (said 'ketchup')
• Contents:
• What's this for? 💻
  • Use-cases
  • Bootstrapping Kubernetes
  • Download k3sup CE (tl;dr)
• Demo of K3sup CE📼
• Usage ✅
  • Pre-requisites for k3sup servers and agents
• K3sup Pro
  • Getting k3sup pro
  • Activating K3sup Pro
  • K3sup plan / apply for automation and large installations
  • Rapid uninstallation / reset with k3sup pro uninstall
  • K3sup pro exec - run a command everywhere
  • K3sup pro get-config - work with an existing cluster
• K3sup Community Edition (CE)
  • 👑 Setup a Kubernetes server with k3sup
  • Checking if a cluster is ready
  • Merging clusters into your KUBECONFIG
  • 😸 Join some agents to your Kubernetes server
  • Use your hardware authentication / 2FA or SSH Agent
  • Create a multi-master (HA) setup with external SQL
  • Create a multi-master (HA) setup with embedded etcd
  • 👨‍💻 Micro-tutorial for Raspberry Pi (2, 3, or 4) 🥧
• Caveats on security
• Contributing
  • Blog posts & tweets
  • Contributing via GitHub
  • License
• 📢 What are people saying about k3sup?
• Similar tools & glossary
• Troubleshooting and support
  • Maybe the problem is with K3s?
  • Common issues
  • Getting access to your KUBECONFIG
  • Smart cards and 2FA
  • Misc note on iptables

What's this for? 💻

This tool uses ssh to install k3s to a remote Linux host. You can also use it to join existing Linux hosts into a k3s cluster as agents. First, k3s is installed using the utility script from Rancher, along with a flag for your host's public IP so that TLS works properly. The kubeconfig file on the server is then fetched and updated so that you can connect from your laptop using kubectl.

You may wonder why a tool like this needs to exist when you can do this sort of thing with bash.

k3sup was developed to automate what can be a very manual and confusing process for many developers, who are already short on time. Once you've provisioned a VM with your favourite tooling, k3sup means you are only 60 seconds away from running kubectl get pods on your own computer. If you are a local computer, you can bypass SSH with k3sup install --local

How does k3sup work?

                          k3sup install
 +---------------------+                  +-----------------------------+
 |                      |    1. SSH       |                             |
 |   Your Laptop /      +---------------->|   Remote Server (VM/RPi)    |
 |   Workstation        |                 |                             |
 |                      |    2. Install   |   +---------------------+   |
 |  +----------------+  |    k3s -------->|   |  k3s (server/agent) |   |
 |  | kubectl        |  |                 |   +---------------------+   |
 |  +----------------+  |    3. Fetch     |                             |
 |  | kubeconfig     |<-+    kubeconfig   |                             |
 |  +----------------+  |                 +-----------------------------+
 |                      |
 +---------------------+        k3sup join
                          +-------------------->+-----------+
                          |                     | Agent 1   |
                          +-------------------->+-----------+
                          |                     | Agent 2   |
                          +-------------------->+-----------+
                                                | Agent n   |
                                                +-----------+

• Step 1: k3sup install → SSH into server, install k3s
• Step 2: kubeconfig → Fetched to your laptop automatically
• Step 3: k3sup join → Add agents to the cluster via SSH
• Step 4: kubectl → Ready to use from your laptop 🚀

Tip: Create clusters on Mac, Linux + WSL2 with K3sup and SlicerVM

Use-cases

K3sup runs from your local machine, without ever having to log into a remote server.

• Bootstrap Kubernetes with k3s onto any VM with k3sup install - either manually, during CI or through cloud-init
• Get from zero to kubectl with k3s on bare-metal, Raspberry Pi (RPi), VMs, AWS EC2, Google Cloud, DigitalOcean, Civo, Linode, Scaleway, and others
• Build a Highly-Available (HA), multi-master (server) cluster
• Fetch the KUBECONFIG from an existing cluster with k3sup pro get-config
• Join nodes into an existing k3s cluster with k3sup join
• Build a massive cluster for automation and scale-out testing using k3sup plan and a JSON file with IP addresses

Bootstrapping Kubernetes

Conceptual architecture, showing k3sup running locally against any VM such as AWS EC2 or a VPS such as DigitalOcean.

Download k3sup CE (tl;dr)

k3sup is distributed as a static Go binary. You can use the installer on MacOS and Linux, or visit the Releases page to download the executable for Windows.

curl -sLS https://get.k3sup.dev | sh
sudo install k3sup /usr/local/bin/

k3sup --help

A note for Windows users. Windows users can use k3sup install and k3sup join with a normal "Windows command prompt".

Demo of K3sup CE📼

In the demo I install Kubernetes (k3s) onto two separate machines and get my kubeconfig downloaded to my laptop each time in around one minute.

1) Ubuntu 18.04 VM created on DigitalOcean with ssh key copied automatically 2) Raspberry Pi 4 with my ssh key copied over via ssh-copy-id

Watch the demo:

Usage ✅

The k3sup tool is a client application which you can run on your own computer. It uses SSH to connect to remote servers and creates a local KUBECONFIG file on your disk. Binaries are provided for MacOS, Windows, and Linux (including ARM).

Pre-requisites for k3sup servers and agents

Some Linux hosts are configured to allow sudo to run without having to repeat your password. For those which are not already configured that way, you'll need to make the following changes if you wish to use k3sup:

# sudo visudo

# Then add to the bottom of the file
# replace "alex" with your username i.e. "ubuntu"
alex ALL=(ALL) NOPASSWD: ALL

In most circumstances, cloud images for Ubuntu and other distributions will not require this step.

As an alternative, if you only need a single server you can log in interactively and run k3sup install --local instead of using SSH.

K3sup Pro

K3sup Pro is available for individuals and commercial use. Review the EULA before downloading or using the software.

The binary name for K3sup Pro is k3sup which is intended to replace the existing k3sup CE binary, if you have it. You'll find all the additional commands behind a new sub-command k3sup pro.

Support for all K3sup Pro users is provided by the Issue Tracker for the K3sup CE repository.

• activate - obtain/refresh a license key. Commercial users just place your key at ~/.k3sup/LICENSE
• plan - take one or more JSON files and generate a YAML plan for a HA installation of K3s
• apply - run the installation in parallel, optionally pre-downloading the K3s binary and copying it via SSH beforehand
• exec - run a command on all nodes in the cluster
• get-config - get a kubeconfig from an existing installation
• uninstall - uninstall k3s from all nodes in the cluster in parallel

The --predownload flag for k3sup pro apply is the first step towards a fully airgapped solution, and reduces bandwidth whilst speeding up installation.

Walkthrough of plan, apply, get-config and exec:

K3sup Pro roadmap

The initial version of K3sup Pro is largely feature-complete, however there are some additional features planned for commercial users:

• Use K3sup Pro Plan/Apply via bastion hosts
• Airgapped installation via initial download of packages on a local machine

Getting k3sup pro

The k3sup pro binary is packaged in a container image, rather than being downloaded via GitHub Releases.

The recommended option is to use K3sup CE to obtain K3sup Pro:

PRO=1 curl -sLS https://get.k3sup.dev | sudo -E sh

Or, if you already have the latest K3sup CE version, it can replace itself:

sudo k3sup get pro

For the k3sup get pro command, you can omit sudo by passing a --path variable to a writeable location by your user.

You can browse specific versions at ghcr.io/openfaasltd/k3sup-pro then pass the --version flag accordingly.

Activating K3sup Pro

Run k3sup pro activate to verify your identity using GitHub.com. You'll only need to do this on your laptop/workstation - machines which will host K3s do not need any additional steps.

Commercial users can place their license key at $HOME/.k3sup/LICENSE and do not need to run k3sup pro activate.

K3sup plan / apply for automation and large installations

The k3sup pro plan command reads a set of JSON files containing your hosts, and will generate a YAML plan file that you can edit to customize the installation.

Example input file:

[
  {
    "hostname": "k3s-server-1",
    "ip": "192.168.129.138"
  },
  {
    "hostname": "k3s-server-2",
    "ip": "192.168.129.128"
  },
  {
    "hostname": "k3s-server-3",
    "ip": "192.168.129.131"
  },
  {
    "hostname": "k3s-agent-1",
    "ip": "192.168.129.130"
  },
  {
    "hostname": "k3s-agent-2",
    "ip": "192.168.129.127"
  }
]

The following will create 1x primary server, with 2x additional servers within a HA etcd cluster, the last two nodes will be added as agents.

k3sup pro plan ./n100.json ./n200.json \
  --user ubuntu \
  --servers 3 \
  --svclb=false \
  --server-extra-args "--disable traefik" \
  --agent-extra-args "--node-label worker=true"

Example plan.yaml file:

version:
    k3s_channel: stable
server_options:
    user: ubuntu
    ssh_port: 22
    k3s_extra_args: --disable traefik
    parallel: 5
    traefik: true
agent_options:
    k3s_extra_args: --node-label worker=true
hosts:
    - name: k3s-1
      role: server
      host: 192.168.138.2
      architecture: x86_64
    - name: k3s-2
      role: server
      host: 192.168.138.3
      architecture: x86_64
    - name: k3s-3
      role: server
      host: 192.168.138.4
      architecture: x86_64
    - name: k3s-agent-1
      role: agent
      host: 192.168.137.2
      architecture: x86_64
    - name: k3s-agent-2
      role: agent
      host: 192.168.137.3
      architecture: x86_64
    - name: k3s-agent-3
      role: agent
      host: 192.168.137.4
      architecture: x86_64

The YAML plan file can be edited and committed to Git for maintenance and future upgrades.

Then when you're ready to install, you can run k3sup pro apply to install in parallel.

The --predownload flag will download the k3s binary to your local machine, then copy it over SSH to each host to speed up the installation.

The --parallel flag sets how many installation steps to run at the same time.

k3sup pro apply \
  --predownload \
  --parallel 10

You can also get hold of your kubeconfig with k3sup pro get-config and then use kubectl to access your cluster.

Merge it into your main KUBECONFIG file:

```bash k3sup pro get-config \ --local-path $HOME/.ku