In this article, you will learn foundational terms and concepts essential for securing Kubernetes clusters. Whether you're a beginner or an experienced professional, this guide covers the critical knowledge required to understand the security dimensions of Kubernetes and methods to identify and detect specific attack techniques. No prior knowledge of Kubernetes is necessary to benefit from the article.
Kubernetes has become a cornerstone in the infrastructure of many organizations looking to streamline their software environments and harness the power of cloud computing. While Kubernetes offers significant advantages, its adoption is not without risks. These include complexities in configuration, management and potential security vulnerabilities. As organizations increasingly rely on Kubernetes, understanding and mitigating these risks is crucial to safeguard their infrastructures against disruptions and security breaches.
Kubernetes security is crucial for protecting both the infrastructure and the applications it orchestrates from unauthorized access and vulnerabilities. Security in Kubernetes is comprehensive, covering various layers including the cluster configuration, network policies, access controls, and resource limitations. Key to enhancing security is the implementation of Role-Based Access Control (RBAC), which limits user and process permissions in line with the principle of least privilege. Network policies are essential for controlling the traffic between pods, ensuring that only necessary communications are permitted, thus minimizing the potential attack surface. Another important mechanism of Kubernetes is Pod Security Standards which are configurable settings that define security policies at the pod level and prevent privilege escalations.
Maintaining a robust security posture requires regular updates and patches, effective logging and monitoring to detect anomalies, and secure management of Secrets and Configurations. These practices collectively help to bolster the security framework of Kubernetes environments.
The article provides useful detection and investigation opportunities for security professionals especially SOC analysts and detection engineers by presenting potential attack techniques and corresponding monitoring strategies. Attack vectors range from unauthorized access by anonymous users to more complex threats such as exploiting vulnerable external-facing applications, secret enumeration, and escaping from a pod to its node using sensitive HostPath volumes. The detection opportunities and suggested investigation steps provide a solid foundation for monitoring and securing your organization's Kubernetes clusters.
If you're already familiar with Kubernetes basics, feel free to skip ahead to the advanced detection strategies that can significantly enhance your security posture.
Kubernetes, an open-source platform for automating containerized applications' deployment, scaling, and management, has transformed how we think about deploying and scaling applications in modern IT environments. As organizations increasingly adopt this powerful orchestration tool, understanding both its foundational concepts and the intricacies of securing a Kubernetes cluster becomes critical. From understanding core concepts to implementing security measures, this guide caters to a diverse audience. Join us as we navigate the complexities of Kubernetes and empower you to safeguard your infrastructure effectively.
A simple use case for leveraging Kubernetes is a company with a website visited by millions of users. To ensure their website remains fast and doesn't crash with high traffic, they use multiple servers. Kubernetes acts like a smart assistant that helps the company manage these servers efficiently. It can automatically adjust resources, fix problems, and scale the servers up during busy times (like a holiday sale) or down when it's quieter, all with minimal human intervention.
In contrast, without Kubernetes (ignoring other orchestration tools), the company would need to manually manage the servers. This would require the IT team to constantly monitor the servers, manually add more servers when needed, and balance traffic to keep the website running smoothly. This approach is slower, more error-prone, and can become overwhelmingly complex as the number of servers grows.
Without further ado, let’s get ourselves familiarized with the technology!
What is Kubernetes?
Kubernetes is an open-source platform designed to automate the deployment, scaling, and management of containerized applications.
Originally designed by Google, the project is now maintained by the Cloud Native Computing Foundation. The name Kubernetes originates from Ancient Greek, meaning 'helmsman' or 'pilot', reflecting its role in steering the ship of modern software development.
There are different components in Kubernetes, here are the crucial components for understanding detection and investigation opportunities:
Three processes every worker node needs to manage its pods:
Complementary to the worker node, a control plane node must have the following 4 processes:
We also would like to familiarize ourselves with some Kubernetes network concepts:
Service - A method for exposing a network application that is running as one or more pods in your cluster. Since pods are ephemeral (i.e. temporary and designed to be short-lived), Service helps to bridge that communication-wise. We can view in the attached screenshot an example of a Service exposing a nginx server of a certain pod using TCP port 80.
Ingress - An API object that manages external access to the services in a cluster, typically HTTP. Ingress may provide load balancing, SSL termination and name-based virtual hosting. In the following screenshot we can see a simple example where an Ingress sends all its traffic to one Service:
Kubernetes configuration concepts:
ConfigMap - An API object used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or configuration files in a volume.
Secret - Object that includes a small amount of sensitive data (password, key, certificate, etc.) It is encoded in base64 and is not encrypted by default. By default, anyone authorized to create a pod in the namespace can access and read any Secret in the namespace.
There are two additional important concepts for us to understand:
Volumes - On-disk files in a container are ephemeral, which presents some problems for non-trivial applications when running in containers. At its core, a volume is a directory, possibly with some data in it, which is accessible to the containers in a pod. Volume storage is parallel & independent to the code (since pods are ephemeral). The volume can be on the node or on a remote host. Kubernetes doesn’t manage data persistence, the user or admin is responsible for backup and saving data.
Namespaces - Namespaces provide a mechanism for isolating groups of resources within a single cluster. Namespaces enable access and resource management for different teams or users.
Now we understand the basic terminology and concepts used in a Kubernetes cluster. In this chapter, we will detail a few of the security concepts of Kubernetes. The main reference of this chapter is the great documentation of Kubernetes.
Control Plane Protection
Control plane protection consists of five main aspects:
Understanding this is key for any security professional dealing with security in Kubernetes.
Transport Security
Traffic in the control plane is encrypted using TLS. By default, the Kubernetes API server listens on port 6443 on the first non-localhost network interface, protected by TLS. Typically the API server serves on port 443.
The API server presents a certificate. This certificate may be signed using a private certificate authority (CA), or based on a public key infrastructure linked to a generally recognized CA.
The cluster’s private certificate authority (if it exists) should be added to the client’s “~/.kube/config” file.
Authentication
After establishing a TLS connection, the HTTP request moves to the authentication step. The cluster creation script or cluster admin configures the API server to run one or more Authenticator modules. We will not dive into Authenticators in this article.
The input to the authentication step is the entire HTTP request; however, it typically examines the headers and/or client certificate.
Authentication modules include client certificates, passwords, plain tokens, bootstrap tokens, and JSON Web Tokens (used for Service accounts).
Multiple authentication modules can be specified, in which case each one is tried in sequence until one of them succeeds.
Authorization
Right after the request passes authentication successfully comes the authorization stage. A request in Kubernetes should encapsulate the username of the requester, the specific action being requested, and the target object influenced by the requested action. Authorization of the request is granted if there is a predefined policy confirming that the user possesses the necessary permissions to perform the indicated action.
Kubernetes supports multiple authorization modules including ABAC (attribute-based access control), RBAC (role-based access control), and Webhook (i.e. Using a remote HTTP service to provide response to the request). Only after exhausting all the authorization modules and none has authorized it, the request is denied.
Admission control
Admission control modules are software components in Kubernetes that have the ability to modify or reject requests. These modules have access to more information than authorization modules, specifically the ability to review the content of the objects being created or modified.
Admission controllers are triggered by requests that aim to create, modify, delete, or connect to (proxy) an object, but they do not intervene in requests that simply read objects. When multiple admission controllers are enabled, they are invoked sequentially.
Unlike authentication and authorization checks, if any admission controller denies a request, the request is immediately rejected without further processing by subsequent controllers.
After successfully passing through all the admission controllers, a request undergoes validation according to the specific routines designated for the API object in question. Upon passing validation, the request is finally persisted to the object store.
Audit
Kubernetes auditing captures a chronological record of security-relevant events, documenting the sequence of actions within a cluster. It logs activity from users, applications interfacing with the Kubernetes API, and operations from the control plane.
Pod Security
Pod Security Policy (PSP) - Was a built-in admission controller that allows a cluster administrator to control security-sensitive aspects of the pod specification. PSP was deprecated in Kubernetes v1.21
Pod Security Standard (PSS) - configurable settings defining the security policies at the pod level and preventing privilege escalations. PSS has 3 modes:
Pod SecurityContext - SecurityContext is a field that can be used in a pod’s settings which defines to the Kubelet how the pod should run (PSS and PSP only provide restrictions). The security settings apply to all containers in the pod.
RBAC
Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization. As mentioned above Kubernetes authorization stage may use RBAC if it is configured. The API objects related to RBAC in a cluster are:
In the example below we can see a Role with pod read & create permissions in the “default” namespace and a RoleBinding granting the role to a user named “bob”:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pods-read-create
namespace: default
subjects:
- kind: User
name: bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-read-create
apiGroup: rbac.authorization.k8s.io
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: pod-read-create
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list", "create"]
With the knowledge we have now about Kubernetes and its security concepts, we can start discussing some common attack vectors used against it. For convenience reasons, we categorize them by cyber kill chain tactics. This is by no means an exhaustive list and is intended to familiarize us with the attacker’s point of view.
Initial Access
Execution
Credential Access
Persistence
*Editor's Note: Congratulations you have made it to the most fun chapter!
This chapter discusses possible detection logics which we found balanced the signal-to-noise ratio. We also describe the recommended investigation steps for each alert generated by the logic to classify the maliciousness of the alert.
The basis for any detection is the logs it runs over. A very attractive bottleneck of logs is the API audit logs. It contains all the information that can be included in an API audit log in the cluster (including the verbs get, list, create, update, delete, patch, and post). Another advantage of API audit logs is that it is provided by Amazon, Google & Microsoft managed Kubernetes Service (EKS, GKE, and AKS respectively).
Let’s continue to the detection & investigation opportunities themselves:
Opportunity 1: Suspicious impersonation attempt by a Kubernetes user
Detects a suspicious impersonation attempt by users in a Kubernetes environment. Impersonation is an API call that lets a user perform an action using the privileges and in the name of another user. In a Kubernetes environment, the normal usage of this feature is limited to specific service accounts performing specific actions. An impersonation attempt that deviates from the normal behavior might point at a threat actor abusing Impersonation privileges for privilege escalation or hiding malicious activity.
Recommendations:
Opportunity 2: Creation or modification of RoleBinding with a highly privileged built-in role
Detects creation or modification of RoleBinding or ClusterRoleBinding to an overly permissive built-in role (e.g. admin, cluster-admin) which can indicate a privilege escalation method. Threat actors can use this operation to gain highly privileged permissions that allow them to execute privileged operations in the cluster (reading secrets, creating pods, etc.).
As mentioned in the previous chapter, Kubernetes uses RBAC (role-based access control) which defines a mechanism for assigning permissions to users/services/groups by their role. The permissions are described in an object called Role (for one namespace) / ClusterRole (for more than one namespace). To bind these Roles (permissions) to a user/service/group we need to create another object called RoleBinding / ClusterRoleBinding and define the role we want to bind and the user/service account that we bind the role to.
Recommendations:
Opportunity 3: Denied request of a Kubernetes Service Account
Detects previously unseen denied API requests originating from a Service Account in a Kubernetes environment. Kubernetes Service accounts follow pre-programmed expected behavior, therefore a denied request from a Kubernetes Service account is suspicious and may indicate deviation from its normal behavior and potentially, compromise.
Recommendations:
Creation of a Pod with a sensitive hostPath volume
Detects creation of a pod with a sensitive volume of type hostPath. A hostPath volume type mounts a sensitive file or folder from the node to the container. If the container gets compromised, the attacker can use this mount to gain access to the node. There are many ways a container with unrestricted access to the host file system can escalate privileges, including reading data from other containers and accessing tokens of more privileged pods.
Recommendations:
Kubernetes command execution in kube-system namespace
Detects execution of commands in the kube-system namespace on a Kubernetes environment. Threat actors try to execute commands on pods in this namespace since it is used for system-level components and contains critical access and information.
Recommendations:
Creation of a Reverse Shell to a Kubernetes Pod
Detects creation of a reverse shell to Kubernetes pod. The reverse Shell enables the creator to execute commands remotely in that pod. Malicious actors often create a reverse shell to a Kubernetes pod as a step towards credential access, access to sensitive data or move laterally in the cluster.
Recommendations:
Kubernetes Secret enumeration
Detects enumeration of Secrets by Get, List, or Watch API calls. Threat actors with access to Secrets can potentially enable lateral movement or privilege escalation and unauthorized access to critical resources. This detection can be implemented using a static threshold (e.g. more than 10 API calls in 1 minute) or with a more sophisticated mechanism using time series analysis or other anomaly detection algorithms.
Recommendations:
Creation or modification of external-facing Kubernetes NodePort Service
Kubernetes NodePort Service exposes a chosen application outside the cluster (i.e. exposes the node the application is running on for a certain port range). Therefore, the creation or modification of external-facing Kubernetes NodePort Service is a possible way for a malicious actor to establish persistence in the cluster.
Recommendations:
Authorization of Kubernetes API request by an unauthenticated user
Detects authorization of Kubernetes API Requests initiated by unauthenticated users. Such events should be generally restricted to specific internal usage of Kubernetes (e.g. livez, healthz, readyz). An allowed request by an unauthenticated user (i.e. system:anonymous) for a different usage might indicate a malicious activity abusing permissions' misconfiguration.
It's recommended to check:
General Investigation Steps
"The Beginner's Guide for Kubernetes Security" is a comprehensive introduction to securing Kubernetes clusters, designed for both beginners and experienced users. It covers essential concepts and components of Kubernetes, the challenges related to its configuration and management, and the potential security risks associated with its use.
Key points covered in the article include:
Overall, the guide acts as an educational resource to not only familiarize users with Kubernetes but also provide them with the necessary tools and knowledge to secure Kubernetes environments effectively.