Kubelet, a vital component in Kubernetes, runs on each node in your cluster. It acts as the field manager, receiving instructions from the Kubernetes API server and ensuring containerized applications run smoothly. Kubelet is responsible for downloading container images, pulling secrets, and launching pods – the basic units containing your application containers. It also monitors container health, restarts failed ones and reports their status back to the API server.
Due to its central role, debugging Kubelet becomes crucial when troubleshooting your Kubernetes environment. In the post, we will continually expand methods on how to debug Kubelet, especially some practices on some real-world use cases.
Debug logs
Like all others program’s debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Similar to debugging Kubelet, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most components in Kubernetes, kubelet uses klog for logging and there are 10 verbosity levels(0-9).
TL;DR: Bumping up to level 5 would satisfy most debugging needs.
| Level | Meaning | Example |
|---|---|---|
| 0 | Always on (Warning, Error, Fatal) | Example |
| 1 | Default level logs when don’t want any verbosity | Example |
| 2 | Most important logs when major operations happen, also the default verbosity level | Example |
| 3 | Extended information | Example |
| 4 | Debug level | Example |
| 5 | Trace level | Example |
| 6 | Display requested resources | Example |
| 7 | Display HTTP request headers | Example |
| 8 | Display HTTP request payload | Example |
By the time, this note was written. In kubelet related code, level 8 was only used pkg/kubelet/prober/prober_manager.go and level 7 was only used in pkg/kubelet/logs/container_log_manager.go. 11 occurrences in level 6 were used, and all of them are not part of workload lifecycle related.
Leave a Reply