# Running K3s in LXC on Proxmox

2021-11-14

It has been a while since I’ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I’ve had success with deploying it on resource-contrained Raspberry Pis in the past. I thought that this time it’d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine’s resources.

## What is K3s?#

K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.

## Configure Proxmox#

This gist contains snippets and discussion on how to deploy K3s in LXC on Proxmox. It mentions that `bridge-nf-call-iptables` should be loaded, but I did not understand the benefit of doing this.

## Disable swap#

There is an issue on Kubernetes regarding swap here. There claims to be support for swap in 1.22, but for now let’s disable it:

    sudo sysctl vm.swappiness=0
    sudo swapoff -a

It might be worth experimenting with swap enabled in the future to see how that might affect performance.

### Enable IP Forwarding#

To avoid IP Forwarding issues with Traefik, run the following on the host:

    sudo sysctl net.ipv4.ip_forward=1
    sudo sysctl net.ipv6.conf.all.forwarding=1
    sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf
    sudo sed -i 's/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g' /etc/sysctl.conf

## Create LXC container#

Create an LXC container in the Proxmox interface as you normally would. Remember to:

- Uncheck `unprivileged container`
- Use a LXC template (I chose a debian 11 template downloaded with pveam)
- In memory, set swap to 0
- Create and start the container

### Modify container config#

Now back on the host run `pct list` to determine what VMID it was given.

Open `/etc/pve/lxc/$VMID.conf` and append:

    lxc.apparmor.profile: unconfined
    lxc.cap.drop:
    lxc.mount.auto: "proc:rw sys:rw"
    lxc.cgroup2.devices.allow: c 10:200 rwm

All of the above configurations are described in the manpages. Notice that `cgroup2` is used since Proxmox VE 7.0 has switched to a pure cgroupv2 environment.

Thankfully cgroup v2 support has been supported in k3s with these contributions:

- https://github.com/k3s-io/k3s/pull/2584
- https://github.com/k3s-io/k3s/pull/2844

## Enable shared host mounts#

From within the container, run:

    echo '#!/bin/sh -e
    ln -s /dev/console /dev/kmsg
    mount --make-rshared /' > /etc/rc.local
    chmod +x /etc/rc.local
    reboot

## Install K3s#

One of the simplest ways to install K3s on a remote host is to use k3sup. Ensure that you supply a valid `CONTAINER_IP` and choose the `k3s-version` you prefer. As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases here.

    ssh-copy-id root@$CONTAINER_IP
    k3sup install --ip $CONTAINER_IP --user root --k3s-version v1.22.3+k3s1

If all goes well, you should see a path to the `kubeconfig` generated. I moved this into `~/.kube/config` so that kubectl would read this by default.

## Wrapping up#

Installing K3s in LXC on Proxmox works with a few tweaks to the default configuration. I later followed the Tekton’s Getting Started guide and was able to deploy it in a few commands.

    $ kubectl get all --namespace tekton-pipelines
    NAME                                               READY   STATUS    RESTARTS      AGE
    pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8      1/1     Running   1 (50m ago)   12h
    pod/tekton-dashboard-6bf858f977-qt4hr              1/1     Running   1 (50m ago)   11h
    pod/tekton-pipelines-controller-69fd7498d8-f57m4   1/1     Running   1 (50m ago)   12h

    NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                              AGE
    service/tekton-pipelines-controller   ClusterIP   10.43.44.245    <none>        9090/TCP,8080/TCP                    12h
    service/tekton-pipelines-webhook      ClusterIP   10.43.183.242   <none>        9090/TCP,8008/TCP,443/TCP,8080/TCP   12h
    service/tekton-dashboard              ClusterIP   10.43.87.97     <none>        9097/TCP                             11h

    NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/tekton-pipelines-webhook      1/1     1            1           12h
    deployment.apps/tekton-dashboard              1/1     1            1           11h
    deployment.apps/tekton-pipelines-controller   1/1     1            1           12h

    NAME                                                     DESIRED   CURRENT   READY   AGE
    replicaset.apps/tekton-pipelines-webhook-8566ff9b6b      1         1         1       12h
    replicaset.apps/tekton-dashboard-6bf858f977              1         1         1       11h
    replicaset.apps/tekton-pipelines-controller-69fd7498d8   1         1         1       12h

    NAME                                                           REFERENCE                             TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
    horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook   Deployment/tekton-pipelines-webhook   9%/100%   1         5         1          12h

I made sure to install Tailscale in the container so that I can easily access K3s from anywhere.

If I’m feeling adventurous, I might experiment with K3s rootless.