Execute an utility Kubernetes pod to debug

In this post, I will show you a little trick to execute an utility Kubernetes pod to debug issues in namespace. This trick will save you a lot of time and work for real; especially when all pods running with minimal setup.

The Problem

First, let’s address the context of the problem. By far, when building images for deployment, I believe most of the time you will have minimal container image setup, in which most of the time is to reduce the size of container image as small as possible. So it’s like using Alpine-based image to build, removing bash, removing all common Linux utilities, ex. curl, wget, ping, ps, tar, time…

Then you’re limited to what you can do to test network across the pods.

Let say, we have this setup, one pod/api and one pod/mariadbwith service layering for database service/mariadb, for some reasons, the pod/api couldn’t connect to service/mariadb, and everything is alpine-based with shell removed.

There we have a problem to debug the network of this setup.

The Solution

The trick, as I can think of, is to spawn another pod that runs inside the same namespace of current context. Imagining that it could be done by applying a new Kubernetes Pod template just to have a pod that can have network access to what we created in the environment.

$ kubectl get pod,svc
NAME               READY   STATUS    RESTARTS   AGE
pod/api            1/1     Running   0          26m
pod/mariadb        1/1     Running   0          26m

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/mariadb        ClusterIP   None         (none)        80/TCP    26m

Let’s execute an utility Kubernetes pod to debug this network setup.

$ kubectl run -it --image busybox:latest utils --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup api.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      api.nginx
Address 1: 172.17.0.5 api.nginx.default.svc.cluster.local
/ # nslookup mariadb.mariadb
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      mariadb.nginx
Address 1: 172.17.0.6 mariadb.nginx.default.svc.cluster.local

Now, we’re inside the utils pod running in the same network, since we start with busybox image, which contains a lot of pre-installed utilities for sysadmin, we can do whatever we want to debug the issue. Well, as in above output, the case works well, so nslookup output looks fine to me. But in case having any problem, it should says so.

To verify current status, execute the first command again:

$ kubectl get pod,svc
NAME               READY   STATUS    RESTARTS   AGE
pod/utils          1/1     Running   0           2m
pod/api            1/1     Running   0          32m
pod/mariadb        1/1     Running   0          32m

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/mariadb        ClusterIP   None         (none)        80/TCP    32m

Yay, pod/utils is the guy we’re talking about.

About the handy command that spawns this pod, we only execute for once, so adding --restart=Never --rm will remove the container after we done with the job.

Conclusion

So, that’s it. I have shown you a handy trick that is to execute an utility Kubernetes pod to debug issues. If you know any other tip, please share with me. I’d like to learn from you.