Feature flags and circuit breakers sound straightforward. In a single-server application, they are. In a distributed system running across multiple Kubernetes clusters in multiple regions, the problem of how do you get a configuration change to all pods simultaneously becomes non-trivial.

This post presents an architecture for dynamic configuration management and circuit breakers in Kubernetes that has run in production on business-critical, high-scale systems.

The Problem

You have microservices deployed across multiple Kubernetes clusters — say, eu-west, us-east, ap-southeast. Each cluster has multiple pods of each service.

You need to:

  1. Flip a feature flag for all pods in all clusters simultaneously
  2. Trip a circuit breaker when a downstream integration starts failing
  3. Restore the circuit breaker when the integration recovers

You can’t just update a database record and have all pods pick it up — there’s no shared state between clusters. You can’t use Kubernetes ConfigMaps as the source of truth — they’re per-cluster, and there’s no built-in sync mechanism.

You need a proper distributed configuration architecture.

The Architecture

The solution has three layers:

Layer 1: Central Config API + Storage

A single, globally accessible Config API (REST) backed by persistent storage (PostgreSQL or similar). This is the single source of truth for all configuration.

Key operations:

  • GET /config/{service}/{environment} — read current config
  • PUT /config/{service}/{environment} — update config
  • GET /config/history/{service}/{environment} — audit log

The Config API is your control plane. Operators interact with it (directly or via a UI) to change flags and configuration.

Layer 2: Consul KV Store Per Cluster

Consul provides a distributed key-value store with watch/notify capabilities. Deploy a Consul cluster per Kubernetes namespace (or per cluster).

Consul is the local config cache for each cluster. Pods watch Consul for changes rather than polling the central API. This gives:

  • Low latency config propagation (sub-second when a key changes)
  • Resilience — if the central API is unavailable, pods continue with last-known config
  • No direct coupling between pods and the central API

Layer 3: Config Service (Reconciler)

A Config Service (a microservice or CronJob) runs in each cluster and reconciles config from the central storage into the local Consul:

Central Config API → Config Service → Consul KV Store → Pods

The Config Service:

  1. Polls the central Config API on a regular interval (e.g., every 30 seconds)
  2. Compares the remote config with what’s currently in Consul
  3. Pushes changes to Consul

For time-sensitive changes (circuit breaker trips), the interval can be reduced or the Config API can push to the Config Service via webhook.

How Pods Consume Config

Pods watch the Consul KV Store for changes to their config key. When the key changes, they update their in-memory config without restarting.

Most languages have Consul client libraries that support this pattern:

import consul

c = consul.Consul(host='consul.production.svc.cluster.local')

# Watch for changes
index, data = c.kv.get('config/sensor-api/production', wait='5m', index=0)
config = json.loads(data['Value'])

When Consul notifies the pod of a change, the pod reloads config immediately.

Circuit Breaker Integration

Here’s where it gets interesting. The circuit breaker is driven by Prometheus alerts rather than in-service logic.

The flow:

  1. Prometheus detects that a downstream integration is degraded (error rate > threshold, latency > threshold)
  2. Prometheus fires an alert to Alertmanager
  3. Alertmanager routes the alert to a webhook endpoint on the Config API
  4. The Config API automatically flips the circuit breaker flag for the affected service
  5. The Config Service propagates the change to all clusters’ Consul stores
  6. All pods pick up the change within seconds
# Prometheus alert rule
groups:
- name: circuit-breakers
  rules:
  - alert: PaymentServiceDegraded
    expr: |
      rate(http_requests_total{service="payment-service",status=~"5.."}[5m])
      / rate(http_requests_total{service="payment-service"}[5m]) > 0.1
    for: 2m
    annotations:
      circuit_breaker_target: "payment-service-integration"
      circuit_breaker_action: "open"

The alertmanager webhook configuration:

receivers:
- name: circuit-breaker-webhook
  webhook_configs:
  - url: 'https://config-api.internal/circuit-breaker/trigger'
    send_resolved: true

When the alert resolves (the integration recovers), Alertmanager fires the send_resolved webhook, which closes the circuit breaker automatically.

This is a metrics-driven, automated circuit breaker — the system heals itself without human intervention.

Resilience Design

A few important resilience decisions in this architecture:

Full Config Objects, Not Patches

When the Config Service pushes to Consul, it always pushes the full config object for a service, not a partial update or patch.

Why: partial updates can leave a service in an inconsistent state if there’s a failure mid-update. Full object pushes are idempotent — if the same object is pushed twice, the result is the same.

Sensible Defaults

Every service must define sensible defaults for all config values. If Consul is unavailable at startup, the service starts with defaults. If a config value is missing from Consul, the service uses the default.

Never let a missing config cause a startup failure.

Defaults as Deployed Config

The Consul config for a service in a fresh cluster starts as the deployed defaults (the same values the service uses if Consul is unavailable). This ensures a new cluster is immediately operational without a config sync cycle.

No Partial Config Updates

The Config API validates the full config object before accepting a write. You can’t write half a config. The schema is versioned, and the API rejects configs that don’t match the current schema.

When to Use This Architecture

This is a relatively complex system. It’s appropriate when you have:

  • Multiple Kubernetes clusters (multi-region or multi-environment)
  • Services that need sub-second config propagation
  • Circuit breakers that need to trip automatically based on metrics
  • Strict audit requirements for config changes

For single-cluster deployments or simpler requirements, Kubernetes ConfigMaps with a restart-or-reload mechanism may be sufficient.

Summary

Operator / Prometheus Alert
          ↓
    Config API (REST) + Storage
          ↓ (via Config Service / reconciler)
    Consul KV Store (per cluster)
          ↓ (watch/notify)
    Pod in-memory config

The key properties:

  • Single source of truth (central Config API)
  • Low-latency propagation (Consul watches)
  • Resilience (Consul as local cache; sensible defaults)
  • Automated circuit breaking (Prometheus → Alertmanager → Config API → Consul)
  • Auditability (all changes logged in Config API)