Secrets Management (External Secrets Operator)

Managing sensitive data such as API keys or database passwords is often one of the most painful parts of running applications on Kubernetes. While the rest of your manifests can live in Git, secrets should not be stored in plain text. This lesson summarizes several approaches to managing those secrets and demonstrates how to use the External Secrets Operator (ESO) with Google Cloud Secret Manager.

Options for handling secrets

1. Manual management – apply Secret objects by hand outside your normal workflow. Easy to start with but hard to audit and prone to mistakes.
2. External Secrets Operator – mirror values from an external store (Google Cloud Secret Manager, Vault, etc.) into Kubernetes. The configuration in Git only references the secret name while the sensitive value stays in the external system.
3. Sealed Secrets / SOPs – encrypt secret manifests so the encrypted files can live in Git and are decrypted in the cluster.
4. Skip Kubernetes secrets – fetch values directly from a provider such as Vault or AWS Secrets Manager from your application at runtime.

The remainder of this lesson focuses on option 2 using ESO on a GKE cluster. The YAML manifests referenced below are available in the course repository.

Installing the operator

Install ESO with Helm (from Taskfile.yaml in the repo):

helm repo add external-secrets https://charts.external-secrets.io
helm upgrade --install external-secrets external-secrets/external-secrets \
  -n external-secrets \
  --values values.yaml \
  --create-namespace \
  --version 0.9.19

values.yaml annotates the service account so it can impersonate a Google Cloud IAM service account via Workload Identity:

serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: external-secrets@<PROJECT>.iam.gserviceaccount.com

Authorizing access to Secret Manager

1. Create a Google Cloud IAM service account and grant it the roles/secretmanager.secretAccessor role.
2. Bind the roles/iam.workloadIdentityUser role so the Kubernetes service account can act as the IAM account.
3. Annotate the Kubernetes service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
  annotations:
    iam.gke.io/gcp-service-account: external-secrets@<PROJECT>.iam.gserviceaccount.com

Defining the secret store and secret

Create a ClusterSecretStore that tells ESO to use Google Cloud Secret Manager:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: gcp-store
spec:
  provider:
    gcpsm:
      projectID: <PROJECT>

Then define an ExternalSecret which maps a value from Secret Manager into a Kubernetes Secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: gcp-store
  target:
    name: external-secrets-example-k8s
    creationPolicy: Owner
  data:
    - secretKey: key-k8s
      remoteRef:
        key: external-secrets-example-gcp

Apply these manifests and ESO will retrieve the value from Secret Manager and create the Kubernetes secret external-secrets-example-k8s. Verify it with:

kubectl get secrets external-secrets-example-k8s -o yaml

Using ESO keeps sensitive information out of Git, leverages your cloud provider's access controls, and still allows you to manage non-sensitive configuration alongside the rest of your manifests.