Single Node Kubernetes Cluster on Raspberry Pi

Introduction

Kubernetes is the most popular container orchestrator and is now a mature project created originally by Google and finally announced as an open source project in 2014.

We’ve been thinking about improving my Kubernetes skills and we thought that setting up my own Kubernetes cluster would be a good oportunity to learn more about this amazing technology. There are a few options out there to run a Kubernetes cluster. You can install Minikube on macOS, Linux or Windows, which is a one node Kubernetes cluster running in a VM on top of a host; use Google Cloud GKE or Amazon EKS; or self host your Kubernetes cluster. We already tried Minikube in the past and we wanted to get a full fledged experience without worrying about account trials, so we decided to adquire the most powerful Raspberry Pi at the moment.

Requirements

  1. Raspberry Pi (4 Model B with 4GB RAM is recommended)
  2. MicroSD card (128GB or greater is recommended)
  3. Keyboard
  4. HDMI cable (micro HDMI to HDMI if you by the Raspberry Pi 4 Model B)
  5. MicroSD adapter

Getting Started

Ubuntu Server

Once you have all the required components you can start setting up you Raspberry Pi with the latest Ubuntu Server image compatible with your Raspberry Pi version.

25/03/2020: We chose Ubuntu 19.10 after trying Ubuntu 18.04.4 LTS and having some issues.

Now you can copy the Ubuntu image into the micro SD card (on MacOS):

  1. Find the SD card mountpoint (e.g. /dev/disk2).
  2. Unmount the SD card.
  3. Copy the image into the SD card.
diskutil list
diskutil unmountDisk /dev/disk2
sudo sh -c 'gunzip -c ~/Downloads/ubuntu-19.10.1-preinstalled-server-arm64+raspi3.img.xz | sudo dd of=/dev/disk2 bs=32m'

The next step is to plug the Raspberry Pi and change the default password. From now on you can access the Raspberry Pi through SSH.

Default username/password for Ubuntu 19.10.1 is ubuntu/ubuntu.

To take the security a step further you can use SSH public key authentication:

  1. Generate keys

    ssh-keygen
    
  2. Copy the public key to Ubuntu Server

    ssh-copy-id ubuntu@<raspberrypi_ip>
    
  3. (Optional) Create a ssh config file

    touch ~/.ssh/config
    

    and copy

    Host raspi
      HostName sergiomartin.dynu.com
      port 2280
      User ubuntu
    

MicroK8s

MicroK8s is a lightweight Kubernetes which is great for hardware with limited resources like Raspberry Pi. They recommend you to have at least 20G of disk space and 4G of memory are recommended.

Installation & configuration:

  1. Install it through a snap command:

    sudo snap install microk8s --classic --channel=1.17/stable
    
  2. (Optional) Add your user to the microk8s group:

    sudo usermod -a -G microk8s $USER
    
  3. (Optional) Create an alias for kubectl. Add to ~/.bash_aliases or ~/.zshrc an alias.

    alias kubectl="microk8s.kubectl"
    

    In my case we are also using ZSH shell so we had to uncoment export PATH=$HOME/bin:/usr/local/bin:$PATH and add export PATH=$PATH:/snap/bin in the ~/.zshrc file.

  4. Edit boot parameters:

    sudo vi /boot/firmware/nobtcmd.txt
    

    and prepend the following:

    cgroup_enable=memory cgroup_memory=1
    

    finally reboot the system:

    sudo reboot
    

    Now if you show the content of /proc/cgroups it should show the memory row with 1 instead of 0.

  5. (Optional) Change default host name:

    sudo hostnamectl --static set-hostname pi401
    
  6. Now you can check if everything is up and running.

    microk8s.status --wait-ready
    

    this should show microk8s is running and a list of add-ons.

    if you run kubectl get nodes you should also be able to see somthing like this:

    NAME    STATUS   ROLES    AGE     VERSION
    pi401   Ready    <none>   7d23h   v1.17.3
    
  7. Install and configure kubectl on your local machine:

    brew install kubectl
    

    create .kube/config if not present and add the following:

    apiVersion: v1
    clusters:
    - cluster:
        insecure-skip-tls-verify: true
        server: https://<raspberrpi_ip>:16443
        name: raspberrypi
    contexts:
    - context:
        cluster: raspberrypi
        user: raspberrypiadmin
        name: raspberrypi
    current-context: raspberrypi
    kind: Config
    preferences: {}
    users:
    - name: raspberrypiadmin
        user:
        password: <admin_password>
        username: admin
    

    You can find the Raspberry Pi IP and cluster port with:

    kubectl cluster-info
    

    Admin password can be found by running:

    kubectl config view
    

    now the current context should be raspberry

    kubectl config current-context
    

Add-ons

MicroK8s does not come with extra features out-of-the-box, however it provides a list of add-ons that can be installed.

Let’s start installing the following add-ons:

microk8s.enable dns dashboard storage ingress registry

check add-ons deployments:

kubectl get deployments --all-namespaces

Deploying an App

Deploying an app on a Kubernestes cluster running on a Raspberry Pi is the same as doing it on any other machine, except for one point. The CPU architecture is ARM rather than x86/x64 by Intel or AMD. Thus, Docker based images you use have to be packaged specifically for ARM architecture, otherwise you will get an error like this:

standard_init_linux.go:207: exec user process caused "exec format error"

If the image contains RPI or ARM in the name or description, it can usually be used for the Raspberry Pi. As a first app we are going to deploy a Java service with the latest and greatest Spring Boot version.

  1. The app will expose an HTTP endpoint to convert a string to uppercase.

    @Slf4j
    @RestController
    @SpringBootApplication
    public class Application {
    
       public static void main(String[] args) {
          SpringApplication.run(Application.class, args);
       }
    
       @GetMapping("/uppercase/{input}")
       public String uppercase(@PathVariable("input") String input) {
          log.info("Converting string to uppercase...");
          return input.toUpperCase();
       }
    }
    

    Spring Boot App Repository

  2. Build the app

    mvn clean install
    
  3. Build Docker image

    docker build -t smartinrub/raspberrypimicrok8sjava .
    

    As we previouly mentioned only ARM images can run on Raspberry Pi. Java Docker image for Raspberry Pi.

    FROM arm64v8/openjdk:11.0.6-jdk-buster AS builder
    WORKDIR target/dependency
    ARG APPJAR=target/*.jar
    COPY ${APPJAR} app.jar
    RUN jar -xf ./app.jar
    
    FROM arm64v8/openjdk:11.0.6-jre-slim-buster
    VOLUME /tmp
    ARG DEPENDENCY=target/dependency
    COPY --from=builder ${DEPENDENCY}/BOOT-INF/lib /app/lib
    COPY --from=builder ${DEPENDENCY}/META-INF /app/META-INF
    COPY --from=builder ${DEPENDENCY}/BOOT-INF/classes /app
    ENTRYPOINT ["java","-cp","app:app/lib/*","com.sergiomartinrubio.raspberrypimicrok8sjava.Application"]
    
  4. Push image to Docker Hub

    docker push
    

    You need to run docker login first.

  5. Create replica sets of the application on Kubernetes

    kubectl apply -f deployment.yaml
    

    deployment.yaml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: spring-boot-demo-deployment
      labels:
        app: spring-boot-demo
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: spring-boot-demo
      template:
        metadata:
          labels:
            app: spring-boot-demo
        spec:
          containers:
          - name: raspberrypimicrok8sjava
            image: smartinrub/raspberrypimicrok8sjava
            ports:
            - containerPort: 8080
    
  6. Create service resource

    kubectl apply -f service.yaml
    

    service.yaml

    apiVersion: v1
    kind: Service
    metadata:
      name: spring-boot-demo-service
    spec:
      selector:
        app: spring-boot-demo
      ports:
        - port: 8080
          targetPort: 8080
    
  7. Create ingress resource

    kubectl apply -f ingress.yaml
    

    ingress.yaml

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: spring-boot-demo-ingress
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - http:
          paths:
          - path: /
            backend:
              serviceName: spring-boot-demo-service
              servicePort: 8080
    

Now you should be able to hit the service at https://<raspberry_pi_ip>/uppercase/hello from your home local network.

Firewall Configuration

It’s important to keep your Raspberry Pi secured so we are going to enable the firewall and set a few rules to allow incoming traffic to our app.

Allow pod traffic:

sudo ufw allow in on cni0 && sudo ufw allow out on cni0

Update the cbr0 bridge interface ufw rules:

sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0

Allow routing:

sudo ufw default allow routed

Allow ssh, http and https, Kubernetes management port:

sudo ufw allow 22
sudo ufw allow 80
sudo ufw allow 443
sudo ufw allow 16443

Enable firewall

sudo ufw enable

Expose the Cluster to the Public Network

  1. Buy or use a free DNS address (e.g. dynu.com)
  2. Install and configure DDClient for your dns provider:

    sudo apt install ddclient
    

    When installing Ubuntu, the locales may not be completely set and we might get something like perl: warning: Setting locale failed. This can be fixed by generating the missing locale e.g. sudo locale-gen en_GB.UTF-8.

    DDClient configuration for Dynu:

    # ddclient configuration for Dynu
    #
    # /etc/ddclient.conf
    daemon=60                                                # Check every 60 seconds.
    syslog=yes                                               # Log update msgs to syslog.
    mail=root                                                # Mail all msgs to root.
    mail-failure=root                                        # Mail failed update msgs to root.
    pid=/var/run/ddclient.pid                                # Record PID in file.
    use=web, web=checkip.dynu.com/, web-skip='IP Address'    # Get ip from server.
    server=api.dynu.com                                      # IP update server.
    protocol=dyndns2
    login=<myusername>                                       # Your username.
    password=<YOURPASSWORD>                                  # Password or MD5/SHA256 of password.
    <MYDOMAIN.DYNU.COM>                                      # List one or more hostnames one on each line.                              
    
  3. Run a deamon to keep our dynamic ip address updated on Dynu:

    /usr/sbin/ddclient -daemon 300 -syslog
    
  4. Now you can hit your app with the chosen domain name https://<domain_name>/uppercase/hello.