Google Cloud takes infrastructure security seriously, and this is the reason why GKE control plane upgrades are applied periodically, and why there is no option to disable this. In my experience, in most cases this happened without any issues. But after a recent upgrade of one of GKE clusters, I started seeing the following warnings in the Google Cloud console.
These warnings caused by some clients that still is trying to use the autoscaling/v2beta2 API version of the Horizontal Pod Autoscaler, which was officially deprecated in 1.26. A quick update of the HPA manifests to use the current autoscaling/v2 API version didn’t solve the problem - the Google Cloud console still shows that the deprecated calls are happening.
So, I started reviewing the GKE audit logs to see if I could determine which client was making these calls. But, all I found was that the logs only captured my own attempts to update the HPA. It turns out that GKE audit logs, by default, include only the “Admin Write” operations that write metadata or configuration information.
In order to review the operations for reading metadata or configuration information/user-provided data, I need to explicitly enable the “Admin Read” and “Data Read” audit log types for the Kubernetes Engine API.
I turned on these logs for a short time (since it adds extra logging costs) and found out that the warnings came from the kube-state-metrics agent, which was part of the kube-prometheus-stack Helm chart we installed.
Updating the agent as recommended in the compatibility matrix resolved the issue and got rid of the warnings about using deprecated API calls.
Even though we didn’t face any major problems this time, having a similar situation in another GKE cluster with an outdated KEDA that scales our production workloads shows why it’s important to keep an eye on Kubernetes cluster, even in managed environments like GKE, and proactively update Kubernetes versions to avoid potential problems that might be caused by deprecated API calls.