Practical debugging methods for Kubelet

Kubelet, a vital component in Kubernetes, runs on each node in your cluster. It acts as the field manager, receiving instructions from the Kubernetes API server and ensuring containerized applications run smoothly. Kubelet is responsible for downloading container images, pulling secrets, and launching pods – the basic units containing your application containers. It also monitors container health, restarts failed ones and reports their status back to the API server.

Due to its central role, debugging Kubelet becomes crucial when troubleshooting your Kubernetes environment. In the post, we will continually expand methods on how to debug Kubelet, especially some practices on some real-world use cases.

Debug logs

Like all others program’s debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Similar to debugging Kubelet, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most components in Kubernetes, kubelet uses klog for logging and there are 10 verbosity levels(0-9).

TL;DR: Bumping up to level 5 would satisfy most debugging needs.

0Always on (Warning, Error, Fatal)Example
1Default level logs when don’t want any verbosityExample
2Most important logs when major operations happen, also the default verbosity levelExample
3Extended informationExample
4Debug levelExample
5Trace levelExample
6Display requested resourcesExample
7Display HTTP request headersExample
8Display HTTP request payloadExample

By the time, this note was written. In kubelet related code, level 8 was only used pkg/kubelet/prober/prober_manager.go and level 7 was only used in pkg/kubelet/logs/container_log_manager.go. 11 occurrences in level 6 were used, and all of them are not part of workload lifecycle related.

Add trace cide

Kubelet with Bpfcc-tools


Further readings

Inotify watcher leaks in Kubelet


Leave a Reply

Your email address will not be published. Required fields are marked *