Kubernetes is hard

wafrat -

It's supposed to be elegant and simplify lots of things, but right now it's just making me depressed.

All this time I've been managing VMs manually. But that means I have to update Linux every once in a while, on each of them. I just went through the process on one of them and it took me 2 hours. See http://blog.wafrat.com/upgrading-ubuntu-from-16-4-to-18-4/.

Minikube

Docker container, composite container, swarm, Kubernetes node? Pod? It's really easy to get overwhelmed by all the new concepts and the different tools available. Anyhow, I'll just skip the explanations and talk about what I tried to do.

I tried to run Kubernetes and Minikube on my Mac. The idea behind it is: Kubernetes administrates clusters. Minikube makes a local cluster that basically runs a VM on my notebook. It lets you easily try Kubernetes locally.

I followed this guide:

Install Minikube
Confirm Installation To confirm successful installation of both a hypervisor and Minikube, you can run the following command to start up a local Kubernetes cluster:Note: For setting the --vm-driver with minikube start, enter the name of the hypervisor you installed in lowercase letters where <drive…
Kubernetes

Since Minikube runs a VM, I need to install a supervisor. It gives me the choice between HyperKit, VirtualBox and VMware Fusion. I've never heard of HyperKit, VMWare advertises for its paid service, and I don't want to dig into the site and find the free version. So I install VirtualBox.

Now it's time to run minikube: 

minikube start --vm-driver=virtualbox

Boom, errors.

It tells me I should install a newer version of VirtualBox.

You might want to uninstall it and reinstall at least version 5.0.12

I am running version 6.1...

At the end of the wall of errors, it tells me I should reboot my notebook. Fine.

I reboot, and I get the same errors. Let's try HyperKit. When landing on the HyperKit github page, I can't find a ready-made installer. Instead it shows how to clone the repo and build from source. Huh? It also says that Docker for MacOS automatically installs it already. Great. So it's already installed on my machine. Let's run it.

anhtuan@Anhs-MacBook-Pro ~ % minikube start --vm-driver=hyperkit  
😄  minikube v1.6.1 on Darwin 10.15.1
✨  Selecting 'hyperkit' driver from user configuration (alternates: [virtualbox])
💥  The existing "minikube" VM that was created using the "virtualbox" driver, and is incompatible with the "hyperkit" driver.
👉  To proceed, either:

    1) Delete the existing "minikube" cluster using: 'minikube delete'

    * or *

    2) Start the existing "minikube" cluster using: 'minikube start --vm-driver=virtualbox'
	
💣  Exiting.
anhtuan@Anhs-MacBook-Pro ~ % minikube delete
🔥  Deleting "minikube" in virtualbox ...
💔  The "minikube" cluster has been deleted.
🔥  Successfully deleted profile "minikube"
anhtuan@Anhs-MacBook-Pro ~ % minikube start --vm-driver=hyperkit
😄  minikube v1.6.1 on Darwin 10.15.1
✨  Selecting 'hyperkit' driver from user configuration (alternates: [virtualbox])
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.81 MiB / 10.81 MiB  100.00% 733.46 KiB
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/anhtuan/.minikube/bin/docker-machine-driver-hyperkit 
    $ sudo chmod u+s /Users/anhtuan/.minikube/bin/docker-machine-driver-hyperkit 


Password:
🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.17.0 on Docker '19.03.5' ...
💾  Downloading kubelet v1.17.0
💾  Downloading kubeadm v1.17.0
🚜  Pulling images ...
🚀  Launching Kubernetes ... 
⌛  Waiting for cluster to come online ...
🏄  Done! kubectl is now configured to use "minikube"
⚠️  /usr/local/bin/kubectl is version 1.14.8, and is incompatible with Kubernetes 1.17.0. You will need to update /usr/local/bin/kubectl or use 'minikube kubectl' to connect with this cluster
anhtuan@Anhs-MacBook-Pro ~ %

So the Kubernetes version installed by Docker for MacOS is not compatible with minikube. Great. One hour in and I have not even run one Kubernetes command. For now, I'll run minikube's kubectl.

% minikube kubectl get nodes
NAME       STATUS   ROLES    AGE    VERSION
minikube   Ready    master   3m4s   v1.17.0

Running ghost on Minikube

I read and ported the instructions at https://kubernetes.io/docs/tutorials/hello-minikube/ to run Ghost.

anhtuan@Anhs-MacBook-Pro ~ % kubectl create deployment hello-node --image=ghost
deployment.apps/hello-node created
anhtuan@Anhs-MacBook-Pro ~ % kubectl get pods
NAME                          READY   STATUS              RESTARTS   AGE
hello-node-8467ff6b44-5xlzt   0/1     ContainerCreating   0          15s
anhtuan@Anhs-MacBook-Pro ~ % kubectl get deployments
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
hello-node   0/1     1            0           23s
anhtuan@Anhs-MacBook-Pro ~ % kubectl get deployments
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
hello-node   1/1     1            1           80s
anhtuan@Anhs-MacBook-Pro ~ % kubectl expose deployment hello-node --type=LoadBalancer --port=2368
service/hello-node exposed
anhtuan@Anhs-MacBook-Pro ~ % kubectl get services

NAME         TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
hello-node   LoadBalancer   10.96.8.166   <pending>     2368:31384/TCP   15s
kubernetes   ClusterIP      10.96.0.1     <none>        443/TCP          6m40s
anhtuan@Anhs-MacBook-Pro ~ % minikube service hello-node
|-----------|------------|-------------|---------------------------|
| NAMESPACE |    NAME    | TARGET PORT |            URL            |
|-----------|------------|-------------|---------------------------|
| default   | hello-node |             | http://192.168.64.2:31384 |
|-----------|------------|-------------|---------------------------|
🎉  Opening service default/hello-node in default browser...

Nice:

It ran the blog but the data on this machine is not persistent. For that we need a volume.

Setting up a volume

I used the instructions at https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/.

I made an empty folder at ~/minikube_data. My yaml file is like so:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/Users/anhtuan/minikube_data"

kubectl apply -f minikube_data.yaml

View information about the PersistentVolume:

anhtuan@Anhs-MacBook-Pro ~ % kubectl get pv task-pv-volume
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
task-pv-volume   1Gi        RWO            Retain           Available           manual                  8s


After you create the Volume, you have to create a Claim for space. So I wrote minikube_data_claim.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Notice how the volume goes from available to bound:

anhtuan@Anhs-MacBook-Pro ~ % kubectl apply -f minikube_data_claim.yaml 
persistentvolumeclaim/task-pv-claim created
anhtuan@Anhs-MacBook-Pro ~ % kubectl get pv task-pv-volume
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                   STORAGECLASS   REASON   AGE
task-pv-volume   1Gi        RWO            Retain           Bound    default/task-pv-claim   manual                  2m17s

Now we set up a Pod that uses that volume. My ghost.yaml is like so:

apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: ghost
      ports:
        - containerPort: 2368
          name: "ghost-server"
      volumeMounts:
        - mountPath: "/var/lib/ghost/content"
          name: task-pv-storage

Now I run it:

anhtuan@Anhs-MacBook-Pro ~ % kubectl apply -f ghost.yaml 
pod/task-pv-pod created
anhtuan@Anhs-MacBook-Pro ~ % kubectl get pod task-pv-pod
NAME          READY   STATUS              RESTARTS   AGE
task-pv-pod   0/1     ContainerCreating   0          9s
anhtuan@Anhs-MacBook-Pro ~ % kubectl get pod task-pv-pod
NAME          READY   STATUS    RESTARTS   AGE
task-pv-pod   1/1     Running   0          23s

Like before, I need a load balancer to expose the port. However, before I was running a deployment. This time I am running a Pod.

anhtuan@Anhs-MacBook-Pro ~ % kubectl expose deployment task-pv-pod --type=LoadBalancer --port=2368
Error from server (NotFound): deployments.apps "task-pv-pod" not found
anhtuan@Anhs-MacBook-Pro ~ % kubectl get deployments
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
hello-node   1/1     1            1           17m

It turns out I can expose pods as well according to:

% kubectl expose pod task-pv-pod --port=2368                          
error: couldn't retrieve selectors via --selector flag or introspection: the pod has no labels and cannot be exposed
See 'kubectl expose -h' for help and examples
anhtuan@Anhs-MacBook-Pro ~ %

After adding a label to the pod like so:

apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
  labels:
    app: my-ghost
...

I run it again:

anhtuan@Anhs-MacBook-Pro ~ % kubectl apply -f ghost.yaml
pod/task-pv-pod configured
anhtuan@Anhs-MacBook-Pro ~ % kubectl expose pod task-pv-pod --port=2368
service/task-pv-pod exposed

But it appears I have not set up the pod correctly:

anhtuan@Anhs-MacBook-Pro ~ % kubectl get services
NAME          TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
hello-node    LoadBalancer   10.96.8.166   <pending>     2368:31384/TCP   29m
kubernetes    ClusterIP      10.96.0.1     <none>        443/TCP          36m
task-pv-pod   ClusterIP      10.96.66.45   <none>        2368/TCP         90s
anhtuan@Anhs-MacBook-Pro ~ % minikube service task-pv-pod
|-----------|-------------|-------------|--------------|
| NAMESPACE |    NAME     | TARGET PORT |     URL      |
|-----------|-------------|-------------|--------------|
| default   | task-pv-pod |             | No node port |
|-----------|-------------|-------------|--------------|
😿  service default/task-pv-pod has no node port

Turns out you can't expose a Pod directly. You have a set up a service.

Although each Pod has a unique IP address, those IPs are not exposed outside the cluster without a Service.
Using a Service to Expose Your App
Objectives Learn about a Service in Kubernetes Understand how labels and LabelSelector objects relate to a Service Expose an application outside a Kubernetes cluster using a Service Overview of Kubernetes Services Kubernetes Pods are mortal. Pods in fact have a lifecycle. When a worker node dies,…
Kubernetes

The reason is that a Pod can die. The service uses Pods. It can re-run pods.

I use this guide (https://kubernetes.io/docs/tutorials/stateless-application/expose-external-ip-address/) and set up a service like this:

apiVersion: v1
kind: Service
metadata:
  name: my-ghost-service
  labels:
    run: task-pv-pod
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: task-pv-pod

And run it again:

anhtuan@Anhs-MacBook-Pro ~ % kubectl get svc 
NAME               TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
hello-node         LoadBalancer   10.96.8.166   <pending>     2368:31384/TCP   38m
kubernetes         ClusterIP      10.96.0.1     <none>        443/TCP          44m
my-ghost-service   ClusterIP      10.96.62.33   <none>        80/TCP           13s
task-pv-pod        ClusterIP      10.96.66.45   <none>        2368/TCP         9m54s
anhtuan@Anhs-MacBook-Pro ~ % minikube service my-ghost-service
|-----------|------------------|-------------|--------------|
| NAMESPACE |       NAME       | TARGET PORT |     URL      |
|-----------|------------------|-------------|--------------|
| default   | my-ghost-service |             | No node port |
|-----------|------------------|-------------|--------------|
😿  service default/my-ghost-service has no node port
anhtuan@Anhs-MacBook-Pro ~ % kubectl apply -f ghost_service.yaml
service/my-ghost-service configured
anhtuan@Anhs-MacBook-Pro ~ % minikube service my-ghost-service  
|-----------|------------------|-------------|--------------|
| NAMESPACE |       NAME       | TARGET PORT |     URL      |
|-----------|------------------|-------------|--------------|
| default   | my-ghost-service |             | No node port |
|-----------|------------------|-------------|--------------|
😿  service default/my-ghost-service has no node port

Googling the error returns only 7 results, none of which are helpful. Alright, that's enough struggling for day.