Setting up Kubernetes for local development using k3d and Docker

Kubernetes has become the defacto choice for running workloads for most microservices. Hence keeping parity with production, it’s wise to use Kubernetes for local development. This article uses k3d distributions of Kubernetes since it’s lightweight and runs on docker. If you are confident running Kubernetes, there are other options like minikube and Docker Desktop.

I initially wrote this article for version 3.x of k3d. The later version supports the local registry out of the box in k3d. If you are interested in setting up the registry outside of k3d, follow along, else skips to the k3d cluster setup.

Pre-requisites

Have following software installed

• Docker for Mac / docker-ce
• k3d
• kubectl

Setting up local docker registry

• On Mac, go to Docker for Mac preference, go to the Docker Engine tab to add the following. If on Debian/Ubuntu edit /etc/docker/daemon.json file to do the same

  {
      "debug": true,
      "experimental": false,
      "insecure-registries": [
          "registry.localhost:5000"
      ],
      "registry-mirrors": [
          "https://registry-1.docker.io"
      ]
  }
  

  Note: Above enabled insecure registry, so you don’t have to have a singed certificates to access your local registry. If an image doesn’t exist in the local registry, then the request is forwarded to docker hub

• Update the host file

  # Added for Kubernetes development
  127.0.0.1 registry.localhost
  ::1       registry.localhost
  #End of section
  

• Restart docker service for above to take effect

• Create a local docker registry

  # Create a folder to hold all the images for the local registry so if you move or update the registry, you don't have to build all the images 
  $ mkdir -p $HOME/registry
  
  # Create docker registry
  $ docker run -d --name registry.localhost --restart=always -v $HOME/registry:/var/lib/registry -p 5000:5000 registry
  

Setting up the k3d cluster

• As the docker registry is outside of the k8s cluster, we need to make the k3d aware of the registry. Make k3d aware of the registry bypassing the details during cluster creation. Create the following file and save at $HOME/k3d-registries.yaml

  $ cat << EOF > $HOME/k3d-registries.yaml
  mirrors:
      "registry.localhost:5000":
         endpoint:
           -  "http://registry.localhost:5000"
  EOF
  

• Create the cluster with the name dev as the name

  $ k3d cluster create dev --port 9000:80@loadbalancer -v $HOME/k3d-registries.yaml:/etc/rancher/k3s/registries.yaml 
  

  Note: Do not use the relative paths for volume

  The above command creates a single node cluster named dev and adds a loadbalancer and ingress controller traefik. Loadbalancer sits outside of the cluster as a docker container and routes traffic through to each node, in this case, just to one node. The above command also should add k8s context to your kubeconfig. If not, follow the command outputs to do the same.

• Connect local docker registry network to k8s network so that k8s can pull images from it. Look at network names using docker network ls

  $ docker network connect k3d-dev registry.localhost
  

  Note: Skip this step if the docker registry is hosted externally and can reach with URL mentioned in $HOME/k3d-registries.yaml

Test the cluster and setup

Test your setup to prove you can successfully push to the registry, and your k8s cluster can pull from it

# Pull a nginx latest image 
docker pull nginx:latest

# Tag the nginx image 
docker tag nginx:latest registry.localhost:5000/nginx:k8s

# Push the tagged image to your local docker registry
docker push registry.localhost:5000/nginx:k8s

# Run a container on your local k8s cluster pulled from local docker registry
kubectl run nginx --image=registry.localhost:5000/nginx:k8s

If you see the successful running of the pod on the k8s cluster with an image pulled from the local docker registry, you have a working setup!