Introduction

democratic-csi implements the csi (container storage interface) spec providing storage for various container orchestration systems (ie: Kubernetes).

The current focus is providing storage via iscsi/nfs from zfs-based storage systems, predominantly FreeNAS / TrueNAS and ZoL on Ubuntu.

The current drivers implement the depth and breadth of the csi spec, so you have access to resizing, snapshots, clones, etc functionality.

democratic-csi is 2 things:

• several implementations of csi drivers
• freenas-nfs (manages zfs datasets to share over nfs)
• freenas-iscsi (manages zfs zvols to share over iscsi)
• freenas-smb (manages zfs datasets to share over smb)
• freenas-nvmeof (manages zfs zvols to share over nvmeof)
• freenas-api-nfs experimental use with SCALE only (manages zfs datasets to share over nfs)
• freenas-api-iscsi experimental use with SCALE only (manages zfs zvols to share over iscsi)
• freenas-api-smb experimental use with SCALE only (manages zfs datasets to share over smb)
• freenas-api-nvmeof experimental use with SCALE only (manages zfs zvols to share over nvmeof)
• zfs-generic-nfs (works with any ZoL installation...ie: Ubuntu)
• zfs-generic-iscsi (works with any ZoL installation...ie: Ubuntu)
• zfs-generic-smb (works with any ZoL installation...ie: Ubuntu)
• zfs-generic-nvmeof (works with any ZoL installation...ie: Ubuntu)
• zfs-local-ephemeral-inline (provisions node-local zfs datasets)
• zfs-local-dataset (provision node-local volume as dataset)
• zfs-local-zvol (provision node-local volume as zvol)
• synology-iscsi experimental (manages volumes to share over iscsi)
• objectivefs (manages objectivefs volumes)
• lustre-client (crudely provisions storage using a shared lustre share/directory for all volumes)
• nfs-client (crudely provisions storage using a shared nfs share/directory for all volumes)
• smb-client (crudely provisions storage using a shared smb share/directory for all volumes)
• local-hostpath (crudely provisions node-local directories)
• node-manual (allows connecting to manually created smb, nfs, lustre, oneclient, nvmeof, and iscsi volumes, see sample PVs in the examples directory)
• containerd-oci-ephemeral-inline (provisions ephemeral rw node-local storage using oci images as a base)
• vhd-ephemeral-inline (provisions ephemeral rw node-local storage using vhd images as a base)
• framework for developing csi drivers

If you have any interest in providing a csi driver, simply open an issue to discuss. The project provides an extensive framework to build from making it relatively easy to implement new drivers.

Installation

Predominantly 3 things are needed:

• node prep (ie: your kubernetes cluster nodes)
• server prep (ie: your storage server)
• deploy the driver into the cluster (helm chart provided with sample values.yaml)

Community Guides

Join us in the Home Operations discord server in #democratic-csi

• https://jonathangazeley.com/2021/01/05/using-truenas-to-provide-persistent-storage-for-kubernetes/
• https://www.lisenet.com/2021/moving-to-truenas-and-democratic-csi-for-kubernetes-persistent-storage/
• https://gist.github.com/admun/4372899f20421a947b7544e5fc9f9117 (migrating from nfs-client-provisioner to democratic-csi)
• https://gist.github.com/deefdragon/d58a4210622ff64088bd62a5d8a4e8cc (migrating between storage classes using velero)
• https://github.com/fenio/k8s-truenas (NFS/iSCSI over API with TrueNAS Scale)
• https://wazaari.dev/blog/truenas-talos-democratic-csi

Node Prep

You should install/configure the requirements for both nfs and iscsi.

cifs

# RHEL / CentOS
sudo yum install -y cifs-utils

# Ubuntu / Debian
sudo apt-get install -y cifs-utils

nfs

# RHEL / CentOS
sudo yum install -y nfs-utils

# Ubuntu / Debian
sudo apt-get install -y nfs-common

iscsi

Note that multipath is supported for the iscsi-based drivers. Simply setup multipath to your liking and set multiple portals in the config as appropriate.

If you are running Kubernetes with rancher/rke please see the following:

• https://github.com/rancher/rke/issues/1846

RHEL / CentOS

# Install the following system packages
sudo yum install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath

# Enable multipathing
sudo mpathconf --enable --with_multipathd y

# Ensure that iscsid and multipathd are running
sudo systemctl enable iscsid multipathd
sudo systemctl start iscsid multipathd

# Start and enable iscsi
sudo systemctl enable iscsi
sudo systemctl start iscsi

Ubuntu / Debian

# Install the following system packages
sudo apt-get install -y open-iscsi lsscsi sg3-utils multipath-tools scsitools

# Enable multipathing
sudo tee /etc/multipath.conf <<-'EOF'
defaults {
    user_friendly_names yes
    find_multipaths yes
}
EOF

sudo systemctl enable multipath-tools.service
sudo service multipath-tools restart

# Ensure that open-iscsi and multipath-tools are enabled and running
sudo systemctl status multipath-tools
sudo systemctl enable open-iscsi.service
sudo service open-iscsi start
sudo systemctl status open-iscsi

Talos

To use iscsi storage in kubernetes cluster in talos these steps are needed which are similar to the ones explained in https://www.talos.dev/v1.1/kubernetes-guides/configuration/replicated-local-storage-with-openebs-jiva/#patching-the-jiva-installation

Patch nodes

since talos does not have iscsi support by default, the iscsi extension is needed create a patch.yaml file with

- op: add
  path: /machine/install/extensions
  value:
    - image: ghcr.io/siderolabs/iscsi-tools:v0.1.1

and apply the patch across all of your nodes

talosctl -e <endpoint ip/hostname> -n <node ip/hostname> patch mc -p @patch.yaml

the extension will not activate until you "upgrade" the nodes, even if there is no update, use the latest version of talos installer. VERIFY THE TALOS VERSION IN THIS COMMAND BEFORE RUNNING IT AND READ THE OpenEBS Jiva. upgrade all of the nodes in the cluster to get the extension

talosctl -e <endpoint ip/hostname> -n <node ip/hostname> upgrade --image=ghcr.io/siderolabs/installer:v1.1.1

in your values.yaml file make sure to enable these settings

node:
  hostPID: true
  driver:
    extraEnv:
      - name: ISCSIADM_HOST_STRATEGY
        value: nsenter
      - name: ISCSIADM_HOST_PATH
        value: /usr/local/sbin/iscsiadm
    iscsiDirHostPath: /usr/local/etc/iscsi
    iscsiDirHostPathType: ""

and continue your democratic installation as usuall with other iscsi drivers.

Privileged Namespace

democratic-csi requires privileged access to the nodes, so the namespace should allow for privileged pods. One way of doing it is via namespace labels. Add the followin label to the democratic-csi installation namespace pod-security.kubernetes.io/enforce=privileged

kubectl label --overwrite namespace democratic-csi pod-security.kubernetes.io/enforce=privileged

nvmeof

# not required but likely helpful (tools are included in the democratic images
# so not needed on the host)
apt-get install -y nvme-cli

# get the nvme fabric modules
apt-get install linux-generic

# ensure the nvmeof modules get loaded at boot
cat <<EOF > /etc/modules-load.d/nvme.conf
nvme
nvme-tcp
nvme-fc
nvme-rdma
EOF

# load the modules immediately
modprobe nvme
modprobe nvme-tcp
modprobe nvme-fc
modprobe nvme-rdma

# nvme has native multipath or can use DM multipath
# democratic-csi will gracefully handle either configuration
# RedHat recommends DM multipath (nvme_core.multipath=N)
cat /sys/module/nvme_core/parameters/multipath

# kernel arg to enable/disable native multipath
nvme_core.multipath=N

zfs-local-ephemeral-inline

This driver provisions node-local ephemeral storage on a per-pod basis. Each node should have an identically named zfs pool created and avaialble to the driver. Note, this is NOT the same thing as using the docker zfs storage driver (although the same pool could be used). No other requirements are necessary.

• https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md
• https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html

zfs-local-{dataset,zvol}

This driver provisions node-local storage. Each node should have an identically named zfs pool created and avaialble to the driver. Note, this is NOT the same thing as using the docker zfs storage driver (although the same pool could be used). Nodes should have the standard zfs utilities installed.

In the name of ease-of-use these drivers by default report MULTI_NODE support (ReadWriteMany in k8s) however the volumes will implicity only work on the node where originally provisioned. Topology contraints manage this in an automated fashion preventing any undesirable behavior. So while you may provision MULTI_NODE / RWX volumes, any workloads using the volume will always land on a single node and that node will always be the node where the volume is/was provisioned.

local-hostpath

This driver provisions node-local storage. Each node should have an identically name folder where volumes will be created.

In the name of ease-of-use these drivers by default report MULTI_NODE support (ReadWriteMany in k8s) however the volumes will implicity only work on the node where originally provisioned. Topology contraints manage this in an automated fashion preventing any undesirable behavior. So while you may provision MULTI_NODE / RWX volumes, any workloads using the volume will always land on a single node and that node will always be the node where the volume is/was provisioned.

The nature of this driver also prevents the enforcement of quotas. In short the requested volume size is generally ignored.

windows

Support for Windows was introduced in v1.7.0. Currently support is limited to kubernetes nodes capabale of running HostProcess containers. Support was tested against Windows Server 2019 using rke2-v1.24. Currently any of the -smb and -iscsi drivers will work. Support for ntfs was added to the linux nodes as well (using the ntfs3 driver) so volumes created can be utilized by nodes with either operating system (in the case of cifs by both simultaneously).

If using any -iscsi driver be sure your iqns are always fully lower-case by default (https://github.com/PowerShell/PowerShell/issues/17306).

Due to current limits in the kubernetes tooling it is not possible to use the local-hostpath driver but support is implemented in this project and will work as soon as kubernetes support is available.

# ensure all updates are installed

# enable the container feature
Enable-WindowsOptionalFeature -Online -FeatureName Containers –All

# install a HostProcess compatible kubernetes

# smb support
# If using with Windows based machines you may need to enable guest access
# (even if you are connecting with credentials)
Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters AllowInsecureGuestAuth -Value 1
Restart-Service LanmanWorkstation -Force

# iscsi
# enable iscsi service and mpio as appropriate
Get-Service -Name MSiSCSI
Set-Service -Name MSiSCSI -StartupType Automatic
Start-Service -Name MSiSCSI
Get-Service -Name MSiSCSI

# mpio
Get-WindowsFeature -Name 'Multipath-IO'
Add-WindowsFeature -Name 'Multipath-IO'

Enable-MSDSMAutomaticClaim -BusType "iSCSI"
Disable-MSDSMAutomaticClaim -BusType "iSCSI"

Get-MSDSMGlobalDefaultLoadBalancePolicy
Set-MSDSMGlobalLoadBalancePolicy -Policy RR

• https://kubernetes.io/blog/2021/08/16/windows-hostprocess-containers/
• https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/

Server Prep

Server preparation depends slightly on which driver you are using.

FreeNAS (freenas-nfs, freenas-iscsi, freenas-smb, freenas-nvmeof, freenas-api-nfs, freenas-api-iscsi, freenas-api-smb, freenas-api-nvmeof)

The recommended version of FreeNAS is 12.0-U2+, however the driver should work with much older versions as well.

The various freenas-api-* drivers are currently EXPERIMENTAL and can only be used with SCALE 21.08+. Fundamentally these drivers remove the need for ssh connections and do all operations entirely with the TrueNAS api. With that in mind, any ssh/shell/etc requirements below can be safely ignored. The minimum volume size through the api is 1G so beware that requested volumes with a size small will be increased to 1G. Also note the following known issues:

• https://jira.ixsystems.com/browse/NAS-111870
• https://github.com/democratic-csi/democratic-csi/issues/112
• https://github.com/democratic-csi/democratic-csi/issues/101

Ensure the following services are configurged and running:

• ssh (if you use a password for authentication make sure it is allowed)
• https://www.truenas.com/community/threads/ssh-access-ssh-rsa-not-in-pubkeyacceptedalgorithms.101715/
• PubkeyAcceptedAlgorithms +ssh-rsa
• ensure zsh, bash, or sh is set as the root shell, csh gives false errors due to quoting
• nfs

• iscsi

• (fixed in 12.0-U2+) when using the FreeNAS API concur