blog

Deploying Secure Elixir Apps with Google Secret Manager

How we improved sharing and managing our application secrets within a team environment.

Samuel Gordalina

@gordalina

Published: Dec 19, 2022

As a company trusted with other companies' valuable content and knowledge, security is a top priority for our engineering team at Slab. We're always looking for ways to improve and one area we identified was how we manage application secrets on a growing team.

Background

When we set out on this journey over two years ago, we were using Elixir 1.11. At the time, the recommended method of managing production secrets was to bundle a config/prod.exs with hardcoded secrets, as this file was included at compile time. We shared this file internally with the engineering team using 1Password.

There were a few drawbacks to using this approach in a Kubernetes environment:

Bundling a file with sensitive data in a docker image is not ideal as it fails the config factor in the twelve-factor app methodology, and if the image is leaked, all secrets have to be considered leaked as well.
We can inject the config/prod.exs as a volume mount, but we would be relying on Kubernetes secrets. We will explain later on why this isn't ideal.
The entire file was shared and updated in its entirety, making it difficult to track and verify which secret was updated. Because of the randomness of values in secrets, it's difficult for teammates to scan or verify they have the latest correct values.

The current recommended method of managing secrets in Elixir applications is to place configurations in config/runtime.exs and have it resolved at runtime via environmental variables.

This is a better approach, as it makes it easier to comply with the twelve-factor app methodology. Relying on environmental variables to store sensitive data creates a credential leakage issue as they can be read by:

Application dependencies
OS applications
Error monitoring tools, e.g., Sentry
Application Performance Monitoring (APM) tools, e.g., AppSignal, DataDog
Container or infrastructure monitoring tools, e.g., Datadog

Some may consider this a good enough compromise, but at Slab, we wanted to achieve a higher standard for our security, especially in a zero-trust security model.

That led us to search for a better solution to manage sensitive data throughout our development, provisioning, and deployment pipelines.

Vault

HashiCorp's Vault is a secure storage system for sensitive data and is widely considered to be the golden standard for secret management.

Its feature set is compatible with our use case, and it would be our first choice for a secret storage service. But we decided not to go with Vault because:

At the time, they only had a self-managed enterprise Vault and no fully managed cloud-hosted offering.
Even their enterprise offering was deployed on Amazon Web Services, whereas Slab hosts all our infrastructure on Google Cloud Platform.
Using Vault, there's also additional complexity in how the application authenticates to it and where that secret is stored.

As a startup, we highly value low overhead and complexity, which led us to search for alternatives.

Google Secret Manager

Fortunately for Slab, Google had recently released Secret Manager in general availability. Like Vault, it's a secure storage system for sensitive data designed for high availability, follows the principle of least privilege, and helps meets audit and compliance requirements, like SOC 2.

Secret Manager, by default, encrypts all secrets at rest using AES-256 and TLS in transit. For an added layer of security, you can use Customer-managed encryption keys (CMEK).

Secrets have versions, which are immutable objects that store sensitive data. A given release may point to a specific secret version or access its latest alias. This makes it easy to manage application secrets, especially when rolling back application configuration.

Since Secret Manager is a managed service, it becomes an excellent choice for startups, and larger organizations, who want an easy and low-maintenance solution for secure secret storage.

Let's see how we can create our first secret using Secret Manager:

# create initial secret
gcloud secrets create DB_PRIMARY_PASSWORD

# create secret's first version with a random password
gcloud secrets versions add test --data-file=- \
  <<< $(openssl rand -base64 32)

# get latest version data
gcloud secrets versions access latest --secret=DB_PRIMARY_PASSWORD

We now have our database password stored in Secret Manager, but how do we leverage it in our application?

Hush

To enable our application to read secrets from Secret Manager, we use Hush, a runtime configuration loader for Elixir applications, which supports multiple providers such as Google Secret Manager.

When hush is configured as a Config Provider in release mode, it resolves the application's configuration during the system boot process and restarts the application with the correct configuration applied to it. Using this method, we can start an Elixir application without writing sensitive data to disk, as all secrets are loaded at runtime.

So how do we configure Hush in our application? First, we add our required dependencies.

# mix.exs

def deps() do
  [
    {:goth, "~> 1.3"},
    {:hush, "~> 1.0"},
    {:hush_gcp_secret_manager, "~> 1.0"},
  ]
end

Secondly, we configure the application release to use Hush as a config provider. When the erlang VM starts, Hush will fetch the configuration from Secret Manager and trigger a restart of the application with the full configuration.

# mix.exs

def project() do
  [
    # ...
    releases: [
      slab: [
        config_providers: [{Hush.ConfigProvider, nil}]
      ]
    ]
  ]
end

If we use Hush configuration in a non-release environment, we'll want Hush to resolve any configuration before our application starts.

# lib/application.ex

defmodule Slab.Application do
  def start(_, _) do
    unless Hush.release_mode?(), do: Hush.resolve!()

    # ...
  end
end

Lastly, we configure hush_gcp_secret_manager to use goth, an Elixir client to authenticate with Google Cloud, and the Google Cloud project to read the secrets from. Make sure you set the GOOGLE_PROJECT_ID environment variable in the application.

# config/runtime.exs

config :hush_gcp_secret_manager,
  project_id: System.fetch_env!("GOOGLE_PROJECT_ID"),
  goth: [name: Hush.Goth]

Now we can use Hush to read configuration from Secret Manager, for example, let's configure our Ecto repo.

# config/runtime.exs

alias Hush.{GcpSecretManager, SystemEnvironment}

if config_env() == :prod do
  config :slab, Slab.Repo,
    hostname: {:hush, SystemEnvironment, "DB_PRIMARY_HOST"},
    password: {:hush, GcpSecretManager, "DB_PRIMARY_PASSWORD"},
    pool_size: {:hush, SystemEnvironment, "SLAB_REPO_POOL_SIZE", cast: :integer}
end

You can read more about Hush and its API on its project page.

Keyless Authentication

A service account is a special type of Google account intended to represent a non-human user that needs to authenticate and be authorized to access data in Google APIs. ― Understanding Service Accounts

These service accounts are used throughout Google Cloud. Some of them are provisioned by default. For example, all Compute VMs are assigned the PROJECT_NUMBER-compute@developer.gserviceaccount.com service account to them as the default service account. This machine can now talk to your Google Cloud APIs using said service account.

You can also create service accounts for specific purposes and only grant them the permissions they need, fulfilling the principle of least privilege.

To interact with Google Cloud APIs, you need a service account and a way to impersonate it. In Google Cloud, there are two ways.

Using a service account key

Unlike normal users, service accounts don't have passwords, they instead use RSA key pairs for authentication. You can generate and download the private service account key and use it to exchange a JWT Bearer token for interacting with Google Cloud. If you used a quick start guide for GCP, there is a good chance you created a service account key file (ends in .json).

On the other hand, service account keys can become a security risk if not managed carefully. Threats related to service account keys include credential leakage, privilege escalation, and information disclosure. The best way to mitigate these threats is to avoid user-managed service account keys whenever possible.

Using Application Default Credentials

Google Cloud API clients that implement Application Default Credentials will always try to locate a service account key, or if the service running the application has an attached service account, can use the metadata server to authenticate to said service account without using a key at all.

One example of this is the compute service account that is attached to a VM by default, any API calls made by an API client running on that VM that use Application Default Credentials will use the compute service account without ever having to use a private key.

Workload Identity in Google Kubernetes Engine

In Google Kubernetes Engine, we can go a step further and use Workload Identity, which enables your Kubernetes cluster to impersonate IAM service accounts to access Google Cloud services.

To do so, we need to have a GKE cluster with workload identity enabled and create a Google Service Account with the roles the app needs, in this case, to access Secrets.

export PROJECT_ID=slab
export GOOGLE_SA=google-sa
export K8S_SA=kubernetes-sa

# create google service account
gcloud iam service-accounts create $GOOGLE_SA

# grant the google service account secret manager role
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$GOOGLE_SA@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

Then we need to create the Kubernetes service account and annotate it to impersonate the Google service account we just created.

# create kubernetes service account
kubectl create serviceaccount $K8S_SA

# allow kubernetes to impersonate the google service account (in default namespace)
gcloud iam service-accounts add-iam-policy-binding $GOOGLE_SA@$PROJECT_ID.iam.gserviceaccount.com \
    --role="roles/iam.workloadIdentityUser" \
    --member="serviceAccount:$PROJECT_ID.svc.id.goog[default/$K8S_SA]"

# annotate kubernetes service account with google's service account
kubectl annotate serviceaccount $K8S_SA \
  iam.gke.io/gcp-service-account=$GOOGLE_SA@$PROJECT_ID.iam.gserviceaccount.com

Now we need to configure our application pods to use the newly create Kubernetes service account.

apiVersion: v1
kind: Pod
metadata:
  name: slab
spec:
  serviceAccountName: $K8S_SA

Any Google client library can access Google Cloud APIs via Workload Identity without using a service account key.

Constantly Learning

Security has always been a top priority at Slab, and we are constantly aiming to push the boundaries for our customers. The use of the best new technologies is one of the ways we strive to meet and exceed their security needs.

We've been using Google Secret Manager and Hush reliably in production for over two years now. We're confident in our choice and usage of them and wanted to share our learnings with the wider Elixir community.