Briefly introduction to kubeconfig files

If you have arrived to this post, probably is because you were looking for information about how to handle your kubeconfig files. But it could be that this is not the case, so let me introduce this file to you:

A kubeconfig file is a file used to configure access to Kubernetes when used in conjunction with the kubectl commandline tool (or other clients).

This file serves the purpose of defining clusters, users and contexts.

Meanwhile a cluster is literally the machine -or set of machines using a shared network- where all the Kubernetes components, capabilities and workloads are configured, a context is a set of access parameters, combination of a cluster, an user and a namespace.

Normally, when we want to access to our Kubernetes cluster through a client as kubectl, the kubeconfig file under ~/.kube/config is automatically detected. When we have a single cluster: we will just put all its configuration on that file.

Managing several clusters

Thing is that normally, in professional environments, we will have access to more than a cluster -probably minimum to three of them if we have one for each development stage and this multiplies in case we're following a CDN approach- and meanwhile I think that having multiple contexts in a same kubeconfig file is a good idea, this normally doesn't apply for different clusters.

Some of the reasons are:

• If we are following proper IAM (Identity and Access Management) guidelines, the access to the different clusters will be different and the parameters for accessing the development environment won't be the same than the ones for accessing the production environment. If we have all of them in the same file, in case that we miss it or share it, we're giving access to all the clusters to a potential attacker.
• Managing the clusters on a single kubeconfig file can be a tedious task, since we have to modify it (manually or using kubectl) in order to delete unused clusters or to add new ones, also we should modify the contexts and users related to that cluster.

So, what could we do?

Working with the KUBECONFIG environment variable

Loading a single Kubeconfig

As we said before, kubectl and other several commandline tools look for the default configuration under ~/.kube/config but there is an environment variable with which we can override this behavior: the KUBECONFIG variable.

If we point to a kubeconfig on that variable, the commandline will load this preferentially, so we could manage different clusters by just referring to the kubeconfig file we want to use:

KUBECONFIG=~/.kube/cluster-dev.yaml kubectl get pods -A

or even

export KUBECONFIG=~/.kube/cluster-dev.yaml
kubectl get namespaces
kubectl get pods -n my-namespace

Meanwhile this solution offers to us a lot of control on which cluster we are using at the moment, it's quite tedious to export the variable to the path location of the file each time we want to change the cluster.

I know what you are thinking! I could set up an alias to each cluster in my shell .rc file and then I would simplify the setup:

alias k8s-dev="export KUBECONFIG='$HOME/.kube/cluster-dev.yaml'"

k8s-dev
kubectl get namespaces
kubectl get pods -n my-namespace

And it's true, you could do that... But meanwhile you are solving the IAM problem of not having all the configurations in a single file, you still have the other one: each time you want to delete a cluster, you will need to modify something manually. In this case, your .zshrc or .bashrc file.

Maybe that's something that you won't need to do on a normal basis and the previous solution is enough for you... but it could be that you're a DevOps consultant with a high rotation of customers and that you would like something more dynamic, or simply you have several clusters to manage for whatever reason. If that's the case, then next solution is for you.

Loading several Kubeconfig files

This is my preferred solution and actually the one I use on my setup: loading dynamically all the different kubeconfig files under a certain folder, one per each cluster.

How does this help us?

Let's suppose we part from the initial case, where we already had all our cluster configurations on a single kubeconfig file, so we need to change the context in order to select the cluster we want to use, right?

Then, why not just letting all those clusters to be loaded (and unloaded!) dynamically to our Kubernetes CLI?

Sounds nice? Then let's see how this is done!

First of all, you need to define a folder where you're going to put all your kubeconfig files. Personally I like to put them under ~/.kube/clusters but it's up to you.

Secondly, you need to define a way of loading all those configurations at once, so for that we are going to modify our .rc file (in my case .zshrc since I'm using ZSH as shell, but in case you're using a different shell like bash, then you should look for .bashrc) and add the following snippet:

export KUBECONFIG=""
separator=""
for cluster in "$HOME/.kube/clusters"/*
do
  if [ -f "$cluster" ]; then
    export KUBECONFIG=$KUBECONFIG$separator$cluster
    separator=":"
  fi
done

This basically will iterate over all the files under your clusters folder and it will attach them to the KUBECONFIG environment variable.

Now for each cluster, you should define one or more contexts with meaningful names, so you remember which cluster and user they're referred to. Save one kubeconfig file for each one of your clusters, using as filename the cluster name and the .yaml extension.

When you finish, you will have something like this:

/Users/dacamposol/.kube/clusters
├── cluster-d1--eu.yaml
├── cluster-d1--us.yaml
├── cluster-i1--eu.yaml
├── cluster-i1--us.yaml
├── cluster-infra--infra.yaml
├── cluster-prod--eu.yaml
└── cluster-prod--us.yaml

0 directories, 7 files

If now you run source .zshrc (or the equivalent for your shell) you will see that you have all the kubeconfig files listed on the environment variable.

Also, when you delete or add a new file in the ~/.kube/clusters folder, you can refresh the contexts to be used by your Kubernetes CLI with the same command (or just creating a new shell session).

Finally, you will be able to see all your loaded contexts with:

kubectl config get-contexts

and choose the desired one with

kubectl config use-context cluster-d1--eu

Enjoy your dynamically managed contexts on Kubernetes!