An Overview of kenv

An Environment File Injector for Kubernetes

Over the past few weeks we’ve been in the process of migrating from Docker Compose to Kubernetes. We wanted to make this transition as easy on our developers as possible which means keeping the development and deployment of applications the same regardless of platforms. One concession we decided to make was migrating our Docker Compose files to the Kubernetes equivilent (in most cases we chose Deployments or DaemonSets). A majority of our Compose files resemble the following:

app:
  image: registry.acme.com/app:${VERSION}
  ports:
    - 8080:8080
  environment:
    - ENVIRONMENT
  env_file:
    - vars/${ENVIRONMENT}.env

The example above relies pretty heavily on variable interpolation and environment files:

  • When running docker-compose, ${VERSION} and ${ENVIRONMENT} will be interpolated with values from the current shell session.
  • The bare ENVIRONMENT key in the environment section is shorthand for ENVIRONMENT=${ENVIRONMENT}. At runtime, the value for the key will be looked up in the current shell session (wherever docker-compose is executed).

Developers have grown accustomed to making changes to applications in various environments through these mechanisms, so it was important that we follow the same pattern in our Kubernetes deployments. As we began migrating our Compose files to Kubernetes we discovered that interpolation of this sort does not exist, but that’s an easy enough problem to solve right before deployment with a bash one-liner (i.e. eval "echo \"$(cat filename)\"") or using a templating engine. However, we weren’t able to find a good substitute for our use of env_file.

kenv: env_file for Kubernetes

In Kubernetes, environment variables can be loaded into a container by adding EnvVar pairs to the container spec. Again, perhaps a templating engine could solve this problem for us, but there is value in being able to run a Kubernetes deployment locally for testing without first rendering it. Out of this need, we made kenv.

kenv works by reading an existing Kubernetes file, in either JSON or YAML, and modifying the container spec to include the variables specified by the user in file(s). For example, if I had the following Kubernetes Deployment file (deployment.yml):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - name: http
          containerPort: 80

And the following environment file (vars.yml):

debug: true
server: acme.com

kenv would inject the environment variables into the deployment like so:

$ kenv -yaml -v vars.yml deployment.yml
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        env:
        - name: debug
          value: "true"
        - name: server
          value: acme.com
        ports:
        - containerPort: 80
          name: http

Pretty simple right? Again, the biggest benefit at this point is that the deployment.yml file is a valid Kubernetes resource by itself, no rendering required.

Continuing with this example, we can create the resource by piping the output of kenv to kubectl:

$ kenv -yaml -v vars.yml deployment.yml | kubectl create -f -
deployment "nginx" created

We can verify that the environment variables were in fact injected by looking at a specific pod (some output omitted):

$ kubectl describe pod/nginx-3419977567-8i05m
Name:		nginx-3419977567-8i05m
... snip ...
Containers:
  nginx:
    Image:		nginx:latest
    Port:		80/TCP
    State:		Running
    Ready:		True
    Environment Variables:
      debug:	true                    # here we see that our variables
      server:	acme.com                # were indeed injected properly

Secrets

This by and large solved our problem, but it also created a bigger security concern. Being able to see plaintext values in the output of a kubectl describe command isn’t ideal for secret data. Fortunately, Kubernetes provides a Secret resource which lets us store key/value pairs as encrypted secrets which we can then access by name in the container spec and ultimately in the container. The biggest difference is the values are never displayed in cleartext in the kubectl commands (although the values are in cleartext in the container itself).

Continuing with the example above, kenv supports specifying certain files as containing secrets, which will in turn be injected as secrets to the container spec. Let’s see an example, continuing on with the example nginx deployment from above (secrets.yml):

username: bob
password: [email protected]

We can specify this as a secret file by using the -s flag when calling kenv (-name is also required to name the created Secret resource):

$ kenv -yaml -v vars.yml -s secrets.yml -name nginx deployment.yml
---
apiVersion: v1
data:
  password: cEBzc3cwcmQ=
  username: Ym9i
kind: Secret
metadata:
  name: nginx
  namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        env:
        - name: debug
          value: "true"
        - name: server
          value: acme.com
        - name: password
          valueFrom:
            secretKeyRef:
              key: password
              name: nginx
        - name: username
          valueFrom:
            secretKeyRef:
              key: username
              name: nginx
        ports:
        - containerPort: 80
          name: http

Here you can see that our normal vars are still injected as before, but now we have our secrets injected as secretKeyRefs. This is telling Kubernetes to perform a lookup to on Secret resource and get the value of a given key and inject it to the container. kenv is doing two things for us here: creating the secret resource and injecting the EnvVars as secretKeyRefs in the container spec.

After applying the changes, we can see that our values are never presented as plaintext in the pod (some output omitted):

$ kenv -yaml -v vars.yml -s secrets.yml -name nginx deployment.yml | kubectl apply -f -
secret "nginx" created
deployment "nginx" configured
$ kubectl describe pod/nginx-1204521676-ktijo
Name:		nginx-1204521676-ktijo
... snip ...
Containers:
  nginx:
    Image:		nginx:latest
    Port:		80/TCP
    State:		Running
    Ready:		True
    Environment Variables:
      debug:	true
      server:	acme.com
      password:	<set to the key 'password' in secret 'nginx'>
      username:	<set to the key 'username' in secret 'nginx'>

Wrapping Things Up

kenv provides a method for injecting environment variables from files as Plaintext K/V pairs, Secrets, and ConfigMaps, and can inject into the following resources:

  • Deployment
  • DaemonSet
  • ReplicaSet
  • ReplicationController

You can download kenv from the Github project site: github.com/thisendout/kenv. Let us know what you think and feel free to report any issues on Github or on Twitter (@thisendout).