Custom resources are the preferred way to extend the K8s API server with a declarative API. They enable us to implement our very own control planes on top of K8s. K8s has performance guidelines and thoroughly investigated scalability thresholds but no guidelines for CRDs are available yet. Our initial attempts to install 1000s of CRDs revealed severe performance issues related to the API server, such as service disruptions and client-side throttling. And this further led to investigations to reveal the root causes of those issues. This talk aims to discuss how one can troubleshoot API server performance issues using profiling tools and to present some real world data that allowed us to pinpoint the root causes of the scaling issues that we initially hit. As the troubleshooting process is explained, the talk will also deliver some insights into the mechanics of CRDs. We would also like to share some tips in successfully getting changes into upstream and moving the ecosystem forward.
Click here to view captioning/translation in the MeetingPlay platform!
