Understanding the Kubernetes Attack Surface
Let’s cover the basics first, the 4C’s is what does that mean. The 4C’s of Cloud Native security are Cloud, Clusters, Containers, and Code. Note: This layered approach augments the defense in depth computing approach to security, which is widely regarded as a best practice for securing software systems.
Here I am scanning to see which ports are open, this is what any hacker would do during recon phase of the attack.
Now we check to see what is inside the docker:
We check the version…
So, it is absolutely important to cover the fundamentals:
Your cluster is as secure as the system running it 🔥
Before you start looking into Kubernetes security specifics you should start with your system running Kubernetes. Go through some guides for securing your OS in general.
Here are some to begin with:
- Red Hat Enterprise Linux 6 Security Guide (Linux in general)
Private topology ⛅
If your infrastructure allows for private IP addresses you should host the cluster in a private subnet and only forward ports that are needed from the outside from your NAT gateway to your cluster. If you’re running on a cloud provider like AWS this can be achieved through a private VPC.
Why? In a private topology all K8s masters and nodes only have private IP addresses assigned. This greatly reduces the risk of exposing anything critical to the public by mistake or lack of knowledge. Also, experience shows that if you need to expose ports explicitly you are more conscious of the potential consequences.
Firewall ports 🔥
This is a general security best practice: Never expose a port, which doesn’t need exposure. IMHO, defining port exposure should be done in the following order:
- Check if you can define a listen IP/interface to bind the service to, if possible 127.0.0.1/lo
- If selectively binding to an IP/interface is not possible, then firewall the port
Kubernetes processes like kubelet are opening a few ports on all network interfaces, which should be firewalled from public access. Those ports may “only” allow to query for sensitive information, but some of them allow straight full access to your cluster.
Port Process Description 4149/TCP kubelet Default cAdvisor port used to query container metrics 10250/TCP kubelet API which allows full node access 10255/TCP kubelet Unauthenticated read-only port, allowing access to node state 10256/TCP kube-proxy Health check server for Kube Proxy 9099/TCP calico-felix Health check server for Calico (if using Calico/Canal) 6443/TCP kube-apiserver Kubernetes API port
Health check ports are no security threat per se stemming from the information they expose, but critical components like the network provider could be DoSed through an exposed health check port, which would affect the whole cluster. Additionally, unknown exploits could potentially endanger security.
Why? Every port exposed to a network poses a potential attack vector. To minimize risk, exposure should be avoided if possible.
Bastion host ☁️
Don’t provide straight public SSH access to each Kubernetes node, use a bastion host setup where you expose SSH only on one specific host from which you SSH into all other hosts. There are quite a few articles on how to do this, for example https://www.nadeau.tv/ssh-with-a-bastion-host/. Also SSH session recording as described in https://aws.amazon.com/blogs/security/how-to-record-ssh-sessions-established-through-a-bastion-host/ can be useful.
For general SSH hardening check Hardening OpenSSH and the OpenSSH chapter in Applied Crypto Hardening by bettercrypto.org.
Why? SSH is a critical service which is under constant attack if exposed to the public. Still today, key-based authentication is not used consistently and everywhere and dictionary attacks or exploits will eventually lead to intrusion. To minimize risk a hardened bastion host is introduced and SSH blocked for public access on all other nodes.
Kubernetes Security Scan with kube-bench 🔥
A very helpful tool to eliminate roughly 95% of the configuration flaws is kube-bench. A master or a node and their control-plane components are checked by applying the CIS Kubernetes Benchmark which results in specific guidelines to secure your cluster setup. This should be a first step before going through any specific Kubernetes security issues or security enhancements.
API settings
Authorization mode & anonymous auth 💥
Some installers like kops will use the AlwaysAllow authorization mode for the cluster. This would grant any authenticated entity full cluster access. Instead, RBAC should be used for role-based access control. To find out what your current configuration is, check the — authorization-mode parameter of your kube-apiserver processes. More information on that topic at https://kubernetes.io/docs/admin/authorization/. To enforce authentication, make sure anonymous auth is disabled by setting — anonymous-auth=false.
Note This doesn’t affect the kubelet authorization mode. The kubelet itself exposes an API to execute commands through which the Kubernetes API can be bypassed completely.
Insecure Port 💥
The insecure port (especially relevant for older Kubernetes releases) is an API port without any kind of protection. No SSL/TLS, no auth, no authz! This port really should be disabled by setting --insecure-port=0. Sometimes it's not possible, because of health check configuration or bootstrapping mechanisms. If that's the case firewall the port from public and internal access. This flag is deprecated since v1.10.
Disable Profiling ☁️
It’s recommended to disable the profiling API endpoint by setting --profiling=false.
Why?
Sensible program or system information can be uncovered by profiling data and because of the amount of data and load induced by profiling your cluster could be DoSed by this feature.
AdmissionController
Add the following plugins to --admission-control=
AlwaysPullImages ☁️
--admission-control=...,AlwaysPullImages
Why?
By default Pods can specify there own image pull policy. Once an image is pulled (even if once pulled from a secured registry with credentials) other Pods could reuse the locally stored image and get access to potentially confidential information. By enabling the AlwaysPullImages policy the controller modifies every new Pod to force the image pull policy to Always, which ensures that credentials need to be provided every time.
DenyEscalatingExec 💥
--admission-control=...,DenyEscalatingExec
Why?
If pods are scheduled with privileged: true, hostPID: true or hostIPC: true it's possible to escalate privileges through attaching to a privileged pod or executing a command in it. The DenyEscalatingExec denies attach and exec for such Pods.
PodSecurityPolicy 💥
--admission-control=...,PodSecurityPolicy
Why?
If PodSecurityPolicy is not enabled, defined Pod Security Policies are not enforced and Pods violating defined policies will still be scheduled.
Warning: Before enabling PodSecurityPolicy you should have Pod security policies already in place or Pods will fail to be scheduled. See Use Pod Security Policies for a basic setup.
As mentioned above, these aforementioned are some fundamentals any kubernetes beginner needs to refer to. There is more. I plan to write another article with more attack vectors and attack commands as samples and prevention techniques.
Sources and kudos to: Simon Pirschel — Kubernetes Consultant & DevOps Specialist for contributing in this github page: https://github.com/freach/kubernetes-security-best-practice
