In this post, I will show you a little trick to execute an utility Kubernetes pod to debug issues in namespace. This trick will save you a lot of time and work for real; especially when all pods running with minimal setup.
First, let’s address the context of the problem. By far, when building images for deployment, I believe most of the time you will have minimal container image setup, in which most of the time is to reduce the size of container image as small as possible. So it’s like using Alpine-based image to build, removing bash, removing all common Linux utilities, ex.
Then you’re limited to what you can do to test network across the pods.
Let say, we have this setup, one
pod/api and one
pod/mariadbwith service layering for database
service/mariadb, for some reasons, the
pod/api couldn’t connect to
service/mariadb, and everything is alpine-based with shell removed.
There we have a problem to debug the network of this setup.
The trick, as I can think of, is to spawn another pod that runs inside the same namespace of current context. Imagining that it could be done by applying a new Kubernetes Pod template just to have a pod that can have network access to what we created in the environment.
$ kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/api 1/1 Running 0 26m pod/mariadb 1/1 Running 0 26m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mariadb ClusterIP None (none) 80/TCP 26m
Let’s execute an utility Kubernetes pod to debug this network setup.
$ kubectl run -it --image busybox:latest utils --restart=Never --rm If you don't see a command prompt, try pressing enter. / # nslookup api.nginx Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: api.nginx Address 1: 172.17.0.5 api.nginx.default.svc.cluster.local / # nslookup mariadb.mariadb Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: mariadb.nginx Address 1: 172.17.0.6 mariadb.nginx.default.svc.cluster.local
Now, we’re inside the
utils pod running in the same network, since we start with
busybox image, which contains a lot of pre-installed utilities for sysadmin, we can do whatever we want to debug the issue. Well, as in above output, the case works well, so
nslookup output looks fine to me. But in case having any problem, it should says so.
To verify current status, execute the first command again:
$ kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/utils 1/1 Running 0 2m pod/api 1/1 Running 0 32m pod/mariadb 1/1 Running 0 32m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mariadb ClusterIP None (none) 80/TCP 32m
pod/utils is the guy we’re talking about.
About the handy command that spawns this pod, we only execute for once, so adding
--restart=Never --rm will remove the container after we done with the job.
So, that’s it. I have shown you a handy trick that is to execute an utility Kubernetes pod to debug issues. If you know any other tip, please share with me. I’d like to learn from you.