Skip to content

Run HA with Leader Election

Use leader election when running more than one replica. Only the leader dispatches alerts; followers stand by.

Enable HA

replicaCount: 2
leaderElection:
  enabled: true
  # Namespace holding the Lease object. Defaults to the release namespace
  # when empty; a shared "system" namespace lets you reinstall the release
  # without losing the lease.
  namespace: kube-system

Apply:

helm upgrade alertkube ./helm --reuse-values \
  --set replicaCount=2 \
  --set leaderElection.enabled=true \
  --set leaderElection.namespace=kube-system

The chart refuses to render replicaCount > 1 unless leader election is enabled.

Follower Behavior

Leadership uses a coordination.k8s.io/v1 Lease with a 30s lease / 20s renew / 5s retry profile (tuned for a workload pod renewing through the API server, unlike the tighter kube-controller-manager defaults):

  • Only the leader dispatches. Followers run the process but do not watch-and-dispatch; they wait to acquire the lease.
  • Followers stay healthy. A follower serves /metrics and /healthz normally - standby is a healthy state, not a failure.
  • /readyz returns 503 on followers until the replica acquires the lease. This is intentional: readiness reflects "am I the active controller," so dashboards and probes can tell leader from standby.

Deployment Strategy

  • Leader election ON → RollingUpdate with maxSurge: 1, maxUnavailable: 0. Leadership transfers to a healthy replica during the rollout, so there is no alerting gap.
  • Leader election OFF (replicaCount: 1) → Recreate. The old pod is torn down before the new one starts, so two instances never overlap and re-fire each other's alerts.

Lease RBAC

The chart adds Lease RBAC in leaderElection.namespace:

  • coordination.k8s.io/leases: get, list, watch, create, update, patch, delete
  • events: create, patch
kubectl get role,rolebinding -n kube-system | grep alertkube

If the chart does not manage that namespace, ensure the ServiceAccount has Lease access.

Verify

  1. Confirm two pods are running and the Lease exists with a holder:

    kubectl get pods -l app.kubernetes.io/name=alertkube
kubectl get lease -n kube-system | grep alertkube

  2. Confirm exactly one pod is the leader via its readiness:

    # leader -> 200, follower -> 503
kubectl exec <leader-pod>   -- wget -qS -O- http://localhost:9090/readyz
kubectl exec <follower-pod> -- wget -qS -O- http://localhost:9090/readyz

  3. Delete the leader pod and confirm the follower acquires the lease within ~15 s, its /readyz flips to 200, and alert dispatch continues without duplicates.

Keep persistence.enabled: true in HA so handovers preserve pending resolves and mute history.

Active/passive vs. active/active (sharding)

Leader election above is active/passive: replicas give you failover, not more throughput - only the leader works. For very large clusters that need to spread the watch/evaluate/dispatch load across replicas, alertkube also supports active/active sharding (v1.2+).

Each replica watches everything but only acts on the objects it owns, where ownership is a stable hash:

owns(object) == fnv32a("kind/namespace/name") mod SHARD_TOTAL == SHARD_INDEX

so at any instant exactly one replica owns a given object - no double-paging. Enable it by giving each replica a distinct index out of a fixed total:

ALERTKUBE_SHARD_TOTAL=3     # number of shards (all replicas)
ALERTKUBE_SHARD_INDEX=0     # this replica's shard (0..TOTAL-1, unique per pod)

Sharding needs a stable per-replica identity

Each replica must get a unique, stable ALERTKUBE_SHARD_INDEX. Run the shards as a StatefulSet (map the pod ordinal to the index via the Downward API / an init step) or as N separate Deployments, one per index. A plain Deployment cannot give replicas stable ordinals. Leave ALERTKUBE_SHARD_TOTAL unset/1 (the default) for the standard single-active-replica model, which is unchanged.

Per-shard identity: Lease and state

Turning sharding on scopes two cluster-wide objects to the shard, automatically:

Object Unsharded Sharded (index i)
Leader Lease alertkube alertkube-shard-<i>
State ConfigMap alertkube-state alertkube-state-<i>

Both are required for correctness, not tidiness:

  • The Lease must be per shard. Every shard runs the full controller body for its own slice of the cluster. If all shards contended for one Lease, exactly one would lead and the other N-1 would watch nothing - and because a leader-election follower reports Ready by design, every pod would stay green while most of the cluster silently stopped being alerted on.
  • The state ConfigMap must be per shard. Each shard's snapshot contains only the alerts it owns, so a shared object means every save overwrites the other shards' mute history and delivery outbox. The visible symptom is a re-paging storm after any restart.

If you set persistence.configMapName explicitly while sharding is on, it must end with this replica's shard index. AlertKube refuses to start otherwise rather than let the shards quietly clobber each other:

persistence.configMapName "alertkube-prod-state" is shared across shards: ...
Give each shard its own object by ending the name with its shard index
(e.g. "alertkube-prod-state-1" for shard 1), or leave persistence.configMapName
empty to have it scoped automatically

Rebalancing

Rebalancing is by rollout: change ALERTKUBE_SHARD_TOTAL and redeploy.

Objects move between shards when the total changes. Any delivery still sitting in a moved object's outbox is dropped on replay by its old owner, not re-delivered, so a rebalance cannot double-page; the new owner re-evaluates the object on its next watch event or resync. Dropped records are counted on alertkube_outbox_replay_foreign_total and logged once per startup.

Cross-shard correlation limitation

With sharding on, each replica's rule engine (count/all correlation rules) observes only its shard's alert stream, so a rule that counts across the whole cluster may under-count. Keep correlation rules on a single active replica (leader election, no sharding) if you rely on them.

The two models compose: use leader election for failover of a single active replica, or sharding for horizontal scale - and because each shard has its own Lease, a shard can itself be a leader-elected pair for failover.

Upgrading from a pre-fix build

Before the shard-scoping fix, the Lease and the state ConfigMap were not shard-scoped. On those builds, running shards under leader election left only one shard active, and running them without it made every shard overwrite the others' persisted state. If you ran sharding on an earlier build, delete the old shared alertkube-state ConfigMap after upgrading: its contents are an arbitrary single shard's partial snapshot.

See Also