Certificate Lifecycle Automation with cert-manager and Vault PKI

Part 8 of the Machine Identity & DevSecOps Series

In a standard Kubernetes cluster, the moment a developer manually generates a self-signed certificate using openssl and mounts it into a pod, the system enters a state of high risk. That certificate has a fixed expiration date, a static private key, and no automated mechanism to detect when the key is compromised or when the validity window closes. The mechanism of modern machine identity management replaces this static state with a dynamic loop: a controller watches a declarative resource, triggers an issuance request against a Certificate Authority (CA), receives the signed artifact, and writes it back to a Kubernetes Secret. This cycle repeats automatically, ensuring that the private key and certificate chain are always valid.

The core actor in this process is the cert-manager controller. It does not issue certificates itself; it acts as a generic client. When you define a Certificate resource, you are essentially writing a request to a specific Issuer. If the Issuer is configured for Let's Encrypt, the controller performs a DNS-01 or HTTP-01 challenge to prove domain ownership. If the Issuer is configured for HashiCorp Vault, the controller authenticates to Vault, requests a dynamic certificate, and receives a signed response. The result is always a Kubernetes Secret of type kubernetes.io/tls, containing the tls.crt and tls.key fields.

Consider a scenario where an organization runs a public-facing API gateway and an internal microservices mesh. The public gateway needs a long-lived, publicly trusted certificate. The internal mesh needs short-lived, internal-only certificates to minimize the blast radius if a node is compromised. We solve this by defining two distinct Issuer resources. The first, letsencrypt-prod, points to the Let's Encrypt production CA. The second, vault-pki-internal, points to a Vault PKI secrets engine mounted within the cluster.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-pki-internal
  namespace: cert-manager
spec:
  vault:
    path: "pki/issue/internal-domain"
    server: "https://vault.internal:8200"
    auth:
      kubernetes:
        role: "cert-manager-role"
        secretRef:
          name: "vault-k8s-auth"

The vault-pki-internal issuer uses the vault field to configure the connection. The path specifies the endpoint in the Vault PKI secrets engine where dynamic certificates are generated. The auth section binds the cert-manager service account to a Vault Kubernetes auth method, allowing the controller to request a token and authenticate without hardcoded secrets. When the Certificate resource references this issuer, the controller sends a JSON request to Vault asking for a certificate valid for the specified dnsNames. Vault's PKI engine generates a short-lived certificate (e.g., 24 hours) and returns it to cert-manager.

This short-lived nature is the primary advantage of the Vault PKI integration over Let's Encrypt for internal traffic. Let's Encrypt certificates are typically valid for 90 days, though durations can vary based on CA policy. Vault PKI certificates can be issued for as little as 1 hour. The mechanism here is that the client (the application pod) does not need to know about the expiration date; it simply reads the Secret mounted at /etc/tls. As long as the Secret is updated, the application sees the new certificate.

The Mechanism of Trust and Rotation

The rotation strategy is governed by the renewBefore field in the Certificate resource. By default, cert-manager attempts to renew a certificate 30 days before its expiration. If you configure renewBefore: 24h, the controller will trigger a renewal attempt 24 hours prior to expiration. For the Vault PKI scenario, if a certificate is valid for only 24 hours, setting renewBefore to 12 hours ensures that the controller initiates the renewal process well before the current certificate expires. The controller then creates a new Secret entry with the new certificate data.

The critical mechanism that prevents downtime during this swap involves the Kubernetes Ingress controller. When the Secret referenced by the Ingress resource is updated, the kube-proxy and the Ingress controller (like Nginx or Traefik) detect the change via the Kubernetes API. They must then be signaled to reload their configuration and begin serving the new certificate immediately. While some controllers watch the volume mount directly, others rely on API events. The private key in the Secret changes, but the Ingress resource definition does not. This atomic update happens at the filesystem level, ensuring that the TLS handshake completes successfully with the new certificate before the old one becomes invalid.

For the public-facing side, the mechanism differs slightly. The letsencrypt-prod issuer uses the ACME protocol. The controller creates a temporary Ingress resource or modifies an existing one to place a challenge file in a specific location. Let's Encrypt's servers access this file via HTTP. If the file is present, the domain ownership is verified, and the CA signs the certificate. The controller then writes the signed certificate to the Secret. Unlike Vault, where the issuer is an internal service, Let's Encrypt is a public third party. This requires the cluster to have outbound connectivity to the internet, whereas the Vault PKI flow can operate entirely within a private network.

Trust Boundaries and Configuration

A common point of failure in this architecture is the caBundle. When using Vault PKI, the client application must trust the root CA that signed the intermediate certificate. If the caBundle in the Certificate resource is incorrect, or if the application does not trust the Vault root, the TLS handshake will fail with a "certificate signed by unknown authority" error. The cert-manager solution is to include the full chain of trust in the Secret. The caBundle field in the Certificate resource is optional; if omitted, cert-manager typically relies on the Issuer to provide the chain. However, it can be used to explicitly specify the CA bundle to verify the issuer's response or to force inclusion in the output Secret if the Issuer does not provide it by default. This ensures that the application only needs to trust the single root CA, and the intermediate chain is provided transparently by the Secret.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-api-cert
  namespace: production
spec:
  secretName: my-api-tls
  duration: 24h # Short duration for Vault PKI
  renewBefore: 12h
  subject:
    organizations:
    - "MyCompany"
  dnsNames:
  - "api.internal.mycompany.com"
  issuerRef:
    name: vault-pki-internal
    kind: Issuer

In this configuration, the duration field explicitly sets the validity period to 24 hours. This overrides the default behavior and forces the Vault PKI engine to generate a short-lived certificate. The renewBefore field ensures that the renewal logic kicks in at the 12-hour mark. If the Vault server is unreachable or the renewBefore window is missed, the certificate expires, and the application stops accepting connections. This is why monitoring the cert-manager logs for renewalFailed events is critical.

The tradeoff here is complexity versus security. Using Let's Encrypt for everything is simpler because it requires no internal infrastructure setup, but it exposes the domain to public validation challenges and uses long-lived keys. Using Vault PKI for everything requires a running Vault cluster, a PKI secrets engine, and careful management of the root CA, but it offers granular control over certificate lifetimes and internal trust. The most robust approach, often seen in DevSecOps pipelines, is a hybrid model. Public endpoints use Let's Encrypt for ease of use and public trust, while internal service-to-service communication uses Vault PKI for short-lived, rotating identities.

Conclusion

When the certificate expires, the Secret is updated. The application pod does not restart. It simply reads the new bytes from the mounted volume. The kernel handles the file descriptor update, and the TLS library in the application (like go's crypto/tls or Java's SSLContext) picks up the new chain. This mechanism of "read-once, update-on-change" is what allows for zero-downtime rotation. Without this, you would need to manually trigger a rolling restart of every pod to load the new certificate, introducing operational overhead and potential service disruption.

The final piece of the puzzle is the ServiceAccount binding. The cert-manager controller runs as a service account in the cert-manager namespace. To issue certificates from Vault, this service account must have a matching role in Vault's Kubernetes auth method. This role defines the policies allowed for the certificate request, such as which paths can be accessed and what TTLs are permitted. If the policy is too restrictive, the controller will fail to authenticate, and the Certificate resource will remain in a Pending state with an error message indicating authentication failure.

In summary, automating certificate lifecycle with cert-manager and Vault PKI shifts the burden of key management from human operators to a declarative system. The mechanism relies on the controller watching a Certificate resource, authenticating to a CA (either public or private), and atomically updating a Secret. This ensures that the private key and certificate chain are always current, reducing the risk of expiration-related outages and minimizing the attack surface of long-lived keys. The choice between Let's Encrypt and Vault PKI depends on whether the traffic is public or internal, but the underlying mechanism of automation remains consistent across both.

Common Pitfalls

Even with a robust architecture, several configuration errors frequently cause deployment failures or security gaps.

1. Incorrect caBundle Configuration: Users often manually paste a CA bundle into the Certificate resource expecting it to force a specific chain, not realizing that the Issuer (Vault) is primarily responsible for generating the full chain. If the bundle is mismatched with the Issuer's internal configuration, clients will reject the certificate. Always verify that the Issuer's output chain matches the application's trust store.
2. Missing renewBefore for Short-Lived Certs: A common oversight is relying on the default 30-day renewal window for certificates with 24-hour validity. If renewBefore is not explicitly set to a value smaller than the certificate duration (e.g., 12 hours), the controller will wait too long to renew, leading to immediate expiration and service outage.
3. ServiceAccount Permission Errors: The Pending state in a Certificate resource is often a sign of RBAC misconfiguration. If the Vault Kubernetes Auth method does not grant the cert-manager ServiceAccount the necessary policies to access the PKI path or set the correct TTLs, the issuance request will silently fail until the policy is corrected.

Practical Takeaways

To master cert-manager and Vault PKI integration, keep these mental models in mind:

• Declarative is Non-Negotiable: Treat certificates as code. Never store them manually. If a Certificate resource exists, the state should be managed entirely by the controller.
• Short-Lived is Better: For internal traffic, prioritize short TTLs (hours) over long ones. The operational cost of rotation is negligible compared to the security benefit of reduced blast radius.
• Trust is Local: Remember that the application pod must trust the root CA. The Secret should contain the full chain so the application only needs to configure one root trust anchor.

FAQ

Q: How does cert-manager handle Let's Encrypt? A: cert-manager acts as an ACME client. It creates temporary Ingress resources to host challenge files (HTTP-01) or creates DNS records (DNS-01) to prove domain ownership. Once verified, Let's Encrypt signs the certificate, and cert-manager stores it in a Kubernetes Secret.

Q: What happens if Vault is down during renewal? A: If the Vault server is unreachable when the renewBefore window triggers, cert-manager will log a renewalFailed event. The existing certificate remains valid until its expiration. However, if the failure persists past the expiration time, the service will stop accepting TLS connections until Vault is restored and the controller re-issues the certificate.

Q: How to configure renewal for short-lived certificates? A: You must explicitly set the renewBefore field in the Certificate spec to a duration shorter than the duration field. For example, for a 24-hour certificate, set duration: 24h and renewBefore: 12h to ensure the renewal starts halfway through the lifecycle.