Part 7: Helm and Environments, Kubernetes Starter

Written August 23rd, 2024 by Nathan Frank

Recap

This article picks up from the sixth article: Managing Secrets with Vault in the Kubernetes Starter series.

Know about using Helm and Environments? Skip to the Full Project Setup.

Helm

Helm is a tool to manage Kubernetes with variables

Consider an application like Wordpress, it's everywhere and people want to run it in containers and on Kubernetes. This allows people to run multiple Wordpress instances on a single Docker instance or Kubernetes cluster. Some people may use base values, while power users need to have access to deeper configuration. They don't need to know the ins and outs of Kubernetes, just learn enough about what they want to configure.

Helm allows us one set of templated declarative Kubernetes yaml files with base values that can be overridden in the form of Helm charts.

Helm can be leveraged for managing different environments

Think about the kinds of differences you might want to have in a DEV env vs an INT/QA/UAT env, STG env, or PROD env:

• Exposed URLs will be different. Consider URLs like: dev.internal.sample.local,qa.internal.sample.local, stg.sample.local, sample.local

• STG and PROD likely have more resources: more CPU, RAM limits, more replicas, max replicas turned up for handling more requests.

• Secrets would be different per environment.

• Service accounts and RBAC might be different.

• Extra services available in lower envs might not be available in higher environments (like Swagger UI, Storybook, or code coverage reports).

• Dependencies (like databases or 3rd party integrations) might be shared in lower envs (to keep costs down) but not in upper envs.

• Lower envs may need to perform unit tests, E2E tests, and have sample test data to support that testing, but possibly not higher envs.

While this could be managed different folders of Kubernetes files (one for each set of environment), a Helm chart could create one set that behaves differently based on input values.

In our case:

1. All envs have different ingress URLs and have different ports for services

2. DEV and QA are set with configMaps to have swagger, while STG and PROD has swagger turned off

3. All envs have different secret files to manage username and password secrets separately

Helm enables multiple environments by using the inputs of the values files and generating the specific Kubernetes objects needed for each installation

Helm charts can live in a chart repository or locally

Helm charts add variables to help with reuse, and are often placed into version control and can be often shared.

Just as container images can used locally or pushed to Docker Hub or a container registry, Helm charts can be used locally or pushed to a remote chart location (like Artifact Hub)

Remote chart registries can be public or run through a series of providers.

Helm charts can install other Helm charts

This allows an entire set of complex applications to be deployed and upgraded in one command, and there's articles on how to do that.

This series is taking a different path as it builds layers of complexity on small concepts and doesn't start with the assumption that people know containers, Kubernetes, and Helm. Helm was used in installing Hashicorp Vault. As we look for flexibility, the same _devops folders with scripts manage all the specific commands needed whether building and running containers in Docker, deploying with Kubernetes declarative files, or deploying with Helm is needed. Normally one wouldn't maintain both Kubernetes declarative files and Helm charts, but this allows us to add in Helm later and not require that the entire solution leverages Helm everywhere.

Helm is also used to install Vault and Vault Secret Operator

Helm can also abstract the complexity away from running and configuring an container/application to just help people understand what is configurable (this may or may not require in-depth knowledge of the application).

We're using Helm charts to install other pieces of the sample application:

• Script shared/vault/_devops/deploy.sh leverages values file shared/vault/_devops/helm/values.vault.yaml

• Script shared/vault-secrets-operator/_devops/deploy.sh leverages values file shared/vault-secrets-operator/_devops/helm/values.vault-operator.yaml

These scripts first add in other Helm chart repositories and update them before attempting to install one.

Local host networking

Some tools like Rancher Desktop share the networking with the host. When this happens running applications locally can conflict with port usage in the single node Rancher Desktop Cluster.

In a full on cluster there would be multiple nodes so there would be less collision and the networking wouldn't be shared making this not a problem.

To solve this locally to be able to run five different instances of sample-node-api we ensure that services have unique ports spec.ports[0].port changing between 30001-30005.

This helps us avoid errors like:

1 node(s) didn't have free ports for the requested pod ports

When running on an actual cluster this isn't an issue because the host networking doesn't exist there, but we need a solution that works both on the cluster and locally for engineers.

Helm and environments hands on

Let's dig into the actual project

Creating a local chart

helm create sample-node-api where sample-node-api is the path/name of the chart to create. You'll notice that there's several files created for you leveraging helm best practices in what to expose.

You could just copy the files you have and only parameterize the values needed, but the best practices note that you'll likely need resources and ingress and services so it stubs them out for you to customize from there.

Breakdown of the custom Helm chart for applications/sample-node-api

Let's compare the kubernetes files at application/sample-node-api/_devops/kubernetes/ to the Helm chart located at: application/sample-node-api/_devops/helm/sample-node-api

• Kubernetes file: 01-sample-node-api.configmap.yaml maps to Helm file: templates/config-map.yaml to set the configuration values for the deployment that are not secrets

• Kubernetes file: 01-sample-node-api.secret.yaml maps to Helm file: templates/secret.yaml to create the base secret with configurable name and default "TBD" values

• Kubernetes file: 01-sample-node-api.vault-auth-static.yaml maps to Helm file: templates/vault-auth-static.yaml to map the role and service name to the default Vault connection

• Kubernetes file: 01-sample-node-api.vault-operator-sa.yaml maps to Helm file: templates/serviceaccount.yaml to configure the service account needed to connect with Vault Secret Operator

• Kubernetes file: 01-sample-node-api.vault-static-secret.yaml maps to Helm file: templates/vault-static-secret.yaml to map the secret to the role to the auth method for the environment

• Kubernetes file: 02-sample-node-api.deployment.yaml maps to Helm file: templates/deployment.yaml to create the deployment with numerous configurable aspects

• Kubernetes file: 03-sample-node-api.service.yaml maps to Helm file: templates/service.yaml to create the service with configurable type

• Kubernetes file: 06-sample-node-api.ingress.yaml maps to Helm file: templates/ingress.yaml to define the URL endpoint that maps to the service

In addition there's other Helm files:

• templates/hpa.yaml to setup horizontal pod scaling

• templates/_helpers.tpl to define helper functions that are used throughout the chart templates

• templates/NOTES.txt to list out the steps to take after the chart is installed

The magic is in the values files

There are files defining the values for each environment like this one:application/sample-node-api/_devops/helm/values.dev.yaml

These allow the configuration to be different per env. Compare the dev values toapplication/sample-node-api/_devops/helm/values.prod.yaml.

Note these files do not contain secrets in them, only references to the secrets to load

Configuration to the nth degree

One could make every possible aspect of a chart configurable and documented, how much time is invested depends on whom might use the chart. It also bears in mind keeping a balance between 100% configuration with what people need to configure.

Reviewing deploy-with-helm.sh

While we have _devops/deploy.sh for applying Kubernetes declarative files, we also have Helm chart scripts for application/sample-node-api that can be installed with _devops/deploy-with-helm.sh.

1#! /bin/bash
2
3cd "$(dirname "$0")"
4echo "****** application shared node api helm setup starting ******"
5
6helm upgrade sample-node-api-dev ./helm/sample-node-api \
7    --install \
8    -f ./helm/values.dev.yaml \
9    --namespace sample-dev --create-namespace --dry-run
10
11echo "****** application shared node api helm setup complete ******"

Of note:

• helm upgrade --install asks helm to upgrade and if it's not there, install it first.

• ./helm/sample-node-api asks helm to use the local chart with that name.

• -f ./helm/values.dev.yaml asks helm to use the specific values values.dev.yaml, we'll have one for each environment.

• --namespace sample-dev --create-namespace specifies that namespace and to create it if it doesn't exist.

• Add --dry-run to see the objects the command would create without actually creating them.

Deploy multiple sample application instances using Helm application/sample-node-api/_devops/deploy-with-helm.sh, but make sure to follow the notes and use application/sample-node-api/_devops/configure.sh (it will need to be run 4x) to reset the secret username and password or the pods will end up in a CrashBackoffLoop, purposefully failing until non "TBD" secrets are provided.

The hosted helm chart for Vault Secret Operator

Vault and Vault Secret Operator are both installed using Helm charts. Once can review how configurable each of these charts are, spoiler: very configurable, because they need to be.

Resilience

All the previous aspects of resilience apply to helm and environments.

Performance

All the previous aspects of performance apply to helm and environments.

Security

All the previous aspects of security apply to helm and environments, but there's likely additional aspects of security that need to be enforced in higher environments. Each team will need to evaluate if these should be reflected in the helm chart/values for configurability. Here we've kept the different environments separate across different namespaces.

Wrap up

We've understood how helm allows us to inject different values into Kubernetes declarative flows to deploy an application multiple times mimicking multiple environments.

Moving along

Continue with the eighth article of the series: Full Project Setup

This series is also available with the accompanying codebase.

Stuck with setup? Refer to full project setup instructions.

Done with the series? Cleanup the project workspace.