Search for a command to run...
Look directly into ETCD, the backbone of your Kubernetes cluster
Passionate engineer with a decade of experience in the cybersecurity industry. Enjoy piecing together solutions involving many moving parts. Big fan of the 'It finally works!' your brain yells when a complex project comes together. Love all things cloud, security, and containerized.
No comments yet. Be the first to comment.
As I type this, I’m scrolling the news and came across yet another vulnerability with an AI tool that removes the tools' safeguards, allowing malicious commands to be run. The rate that these vulnerabilities are occurring is a bit scary but unsurpris...
A simple example of Go modules and binaries
The AWS Documentation is vague on this subject, so this guide will detail the relationship between SecurityHub and Config. It's assumed that knowledge of what SecurityHub and AWS Config are is present. What will follow is based on AWS best practices....
AWS KMS is one of those services that works tirelessly in the background but doesn't get a lot of love. It's common to automatically encrypt every service as part of development and infrastructure-creation workflow, and this usually happens through K...
What is etcd?
ETCD is the distributed key/value database that works in tandem with Kubernetes deployments to hold vital information about the cluster. The security implications of allowing open access to it are vast and can leave your data exposed to bad actors.
ETCD does not operate like other components of the Kubernetes cluster, in that it might not be deployed as a pod, but instead runs as a process across the control plane nodes. Therefore, it is managed completely differently than other components because of the critical nature of it. A system can be rebuilt if etcd is still intact, however a loss of it's data could mean a complete loss of critical data.
Minikube
Minikube is a test environment for Kubernetes that runs usually as a docker container on localhost. It is essentially a 1-node Kubernetes deployment and playground, and is by far the most popular option for development. Because it is a docker container, it's important to know how exactly Minikube is running on your localhost. You can grab it's IP by running:
minikube ip
This prints out the IP of the docker container, as well as the port Minikube is listening on. It is this you will connect to. Once you have minikube up and running, connect into the container via Docker exec:
docker exec -it minikube -- /bin/bash
You now have a shell into the container. You can view the various processes involves with the Kubernetes control plane (since it all runs on this node) by checking the processes and grepping for various services.
ps -aux | grep etcd
ps -aux | grep api
You can view the output and see that kube-apiserver is running with a bunch of command line arguments. A few of these arguments are the cert files required to connect. To connect to etcd, you need following:
CA cert
Server cert
Server key
These are all located in the /var/lib/minikube/certs/etcd directory.
cd /var/lib/minikube/certs/etcd
Minikube runs as an Ubuntu base image and therefore requires you to use apt to install packages. The package we install is etcdctl, the etcd client that connects to the etcd backend.
sudo apt-get update -y
sudo apt-get install -y etcd-client
Now from the etcd certs directory, running the following to test connectivity"
ETCDCTL_API=3 etcdctl --cacert ca.crt --cert server.crt --key server.key --endpoints https://127.0.0.1:2379 get /registry/ --prefix --keys-only
This does a few things:
Sets the etcd-client to the correct API version based on the current etcd version
Sets the appropriate certs and key flags as mentioned before
Tells the etcd-client which endpoint to connect to, in this case, our etcd cluster
Calls a get against the /registry/ URI
Output just the keys (remember, etcd is a K/V store)
This should output a list of various keys in URI format, and they look pretty familiar. For example, you can list all current pods by running:
ETCDCTL_API=3 etcdctl --cacert ca.crt --cert server.crt --key server.key --endpoints https://127.0.0.1:2379 get /registry/ --prefix --keys-only | grep pods/default
Clearly this data is valuable to Kubernetes, and even more to an attacker. This is the Kubernetes backbone, and it's great to actually 'see' where those yaml deployments are stored.
This ability to basically curl the etcd endpoint gives us the ability to be a bit more malicious, where we can view secrets and service-account tokens in plaintext. We cause severe data confidentiality and integrity issues here, but we can also affect the availability of the application by wreaking havok on the cluster. You can perform any number of operations using the etcd API, including writing bogus data or just deciding to completely wipe out the cluster.
Summary
This was a simple demonstration of how to connect locally to your Minikube cluster and view the 'meat' of the Kubernetes cluster. All the deployments, applies, deletes and edits all end up reading/writing to this database. This is why securing your Kubernetes cluster starts with the etcd deployment and ensuring that things like encryption and access-control are firmly in place.