Let’s start an adventure to the bottom of the risk management controls. If you aren’t sure where to start with your defense in depth controls, this isn’t a bad start while you learn at each step of the way. One can almost could think they would have a secure production workload by reaching Ring 2. But many would disagree. In the style of Dante’s Inferno, let’s begin down the two paths in parallel; accessing kubernetes and kubelets via APIs and the kubelet binary accessing the local node and container engine.

The first defense depth ring of security is number 5.

Ring 5

1. Setup TLS on APIServer

2. Limit network access

3. No more all / 0.0.0.0 rules

   1. Enable 22 and 6443 from your external management systems

   2. Verify Read / Write API services, API Server, and etcd are not available from external addresses other than management IPs

   3. Ports 4149, 10255, 10256, 9099, 2379, 8080, 10250, 6443

4. Setup governance limits for launching too many pods, pods consuming too much CPU / RAM, disk space, and / or filesystem nodes

Descending one level deeper into the inferno leads us to 4

Ring 4

1. If not already, enforce Pods governance limits

2. Separate workloads into distinct namespaces

3. Determine how multi-tenancy will operate

4. Multiple etcd?

   1. Multiple controllers?

5. Enforce namespace level governance limits

6. Think about the following threats and determine if warranted to mature to the next defense in depth ring

7. Application tampering

   1. Metrics scaping

   2. Metadata API abuse

   3. Tiller access

   4. Etcd access

   5. Dashboard / Heapster / about the following risks

   6. Service account token abuse

   7. Kubelets exploits and 0days

   8. Established footholds pivoting out of the cluster(s) into underlying infrastructure, systems, and services

Escaping level 4 takes us deeper to level 3.

Ring 3

1. Restricted management (heapster, dashboard, etc..) interfaces disabled or setup to use oidc / authz n / or constantly rotating certificates for access

2. RBAC

3. Entity specific kubeconfigs

4. Kubelet authz/n

5. Admission controllers

6. Logging all the things

7. Etcd TLS

8. Tiller TLS

9. Service mesh policies and CNI management

10. Consider malicious container images, multi-tenant governance challenges, untrusted containers operating in a trusted cluster, and / or container escapes into the cluster(s) services and / or into the underlying infrastructure, systems, and services

Sliding out of 3 into 2. It is getting colder.

Ring 2

1. Production grade admission controllers

2. Egress / ingress filtering full OSI model

3. Vulnerability and security linting detection with automated remediation

4. Containers

   1. Workloads

   2. Clusters

   3. Configs

   4. Behaviors

5. PodSec policies

6. Encrypted etcd

7. Secrets management not Kubernete’s secrets handler

8. Workload and cluster instrumentation such as Falco, twisted, lacework, etc...

9. Container image signing trusts and enforcement

Frozen in Ring 2, it is hard to muster the energy to descend into the deeper defense in depth risk management control hell. Looking over the edge, one may see additional defense in depth rings include secure enclaves, app armor, selinux automated policy generation with corresponding enforcement deployment, and workload isolation techniques with self-healing algorithms incorporating dynamic pod policy resource limits based upon current and projected workload usage patterns.

Looking over the horizon, one is well suited to rotate all secrets’ and certificates’ lifetimes. Setting extremely short lifetimes via rotation greatly limits the liability any one exposed certificate or secret. For some use cases, one may choose to revoke the secret or certificate. For instance, once bootstrapping a service with a temporary Shamir secret, revoke the secret or remove all privileges to the particular Shamir secret.

Reaching into your bag of holding, you notice its’ contents are bare. One would ensure the audit logger and all audit trails tracking critical system events are logged and recorded in a non-repudiated manner that would hold up in any legal proceedings.

You waive your wand wildly in the air and open a crack to the InBetween. You notice there is a book you can reach. Opening the book, you observe metadata about the world you exist within. It is best you seal off that book and any privileges to break out of reality and into the InBetween. For underlining infrastructure as a service providers, that may include removing access or hindering access to the infrastructures’ various metadata services, its’ APIs, and the separation of duties-enforced, least privileged permissions to the infrastructure. By default, they are accessible and will contain secrets which may allow one to escalate within, to, or out of the clusters and / or IaaS account(s.)