Overview

Your cluster is suspected of a breach. Its’ integrity is called into question. Or maybe errant automation ran amuck and took down the cluster by failing to allocate nodes and pods?

• What happened?

• Whom executed the operation?

• Where was the operation?

• From where? To where?

• On what objects or records were affected?

Kube apiserver provided a non-repudiated set of time stamped logs documenting all critical system events and activities. I love digging through this detective control (the audit trails) to keep an eye on my clusters’ well being and traceability. Better yet, you can write you own policy to log every time bob logs and note with a different severity level. Or maybe you don’t care about certain IPs conducting operations? Or maybe you wish to normalize events? Many, many, many flags exist to handle rotation, log lifecycle, and metadata. Personally, I prefer to use web hooks so my SOAR tooling can immediately respond instead of the 30 second to 15 minute lag common in many log management solutions.

Before we saw the kubelet api server deferring its’ web hooks to the api server. The best part about this delegation is that the webhooks’ requests are evaluated before actually executing the requests. I am not familiar with many who would not prefer proactive controls instead of reactive controls. The two use cases commonly used are the ImagePolicy webhook and the ValidationAdmission webhook. If you do not purchase a commercial controller or utilize a popular controller, one may utilize those two hooks to ensure only trusted images and those that are deemed secure are allowed to run.

The web hooks and many other services will ultimately rely upon API calls to perform authorization checks. Depending on the Ring achieve, one will find themselves merging RBAC, Node, and NodeRestriction policies to make authorization decisions. Separation of duties is the best way to handle these complex and nuanced policies. Please keep in mind there are indirect operations for many permissions. Not allowing one to create a pod - great. Easily bypassed by granting someone the privileges to create a deployment - indirectly allows one to create pods. The authentication mechanism I prefer to utilize is Auth0, OIDC, or x.509. Each has their own merits and your mileage may vary.

After reviewing Shodan’s dataset for Kube apiservers publicly exposed in Aliyun, I STRONGLY recommend one implements and enforces TLS / HTTPS for all apiserver traffic. Also do not allow the public Internet to establish connections with the apiserver. Many didn’t have eventrate limits set allowing one to DDOS the apiserver with little resources and bandwidth.

CIS Benchmark

1.2 API Server

1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual)

1.2.2 Ensure that the --basic-auth-file argument is not set (Automated)

1.2.3 Ensure that the --token-auth-file parameter is not set (Automated)

1.2.4 Ensure that the --kubelet-https argument is set to true (Automated)

1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated)

1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated)

1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)

1.2.8 Ensure that the --authorization-mode argument includes Node (Automated)

1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated)

1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) 76

1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated)

1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual)

1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual)

1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated)

1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated)

1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set (Automated)

1.2.17 Ensure that the admission control plugin NodeRestriction is set (Automated)

1.2.18 Ensure that the --insecure-bind-address argument is not set (Automated)

1.2.19 Ensure that the --insecure-port argument is set to 0 (Automated)

1.2.20 Ensure that the --secure-port argument is not set to 0 (Automated)

1.2.21 Ensure that the --profiling argument is set to false (Automated)

1.2.22 Ensure that the --audit-log-path argument is set (Automated)

1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)

1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)

1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)

1.2.26 Ensure that the --request-timeout argument is set as appropriate (Automated)

1.2.27 Ensure that the --service-account-lookup argument is set to true (Automated)

1.2.28 Ensure that the --service-account-key-file argument is set as appropriate (Automated)

1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated)

1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated)

1.2.31 Ensure that the --client-ca-file argument is set as appropriate (Automated)

1.2.32 Ensure that the --etcd-cafile argument is set as appropriate (Automated)

1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Manual)

1.2.34 Ensure that encryption providers are appropriately configured (Manual)

1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual)