Health Checks
Tetrate Global Load Balancer monitors the health and readiness of TSB Gateways that have been annotated with the name of a Gateway set gslb.tetrate.io/name: edge-internal or similar.
The Tetrate GSLB controller does not send its own synthetic HTTP or TCP probes at the Edge Gateway's public IP address. Instead, it determines health from signals already available inside the cluster, combines them into a single healthy/unhealthy verdict per Gateway, and then shares that verdict with every other cluster and region by publishing (or withholding) the Gateway's address in a DNS TXT record, as described in Propagating Health Between Clusters and Regions.
What is Monitored
For each Kubernetes Service that carries a gslb.tetrate.io/name annotation, the controller checks:
-
Endpoint readiness: whether the Service has at least one
Readyendpoint, as reported by KubernetesEndpointSlices. If Kubernetes considers every backing Pod not-ready (for example, because Pods are still starting, failing readiness probes, or have all been terminated), the Gateway is considered unhealthy. -
External address: whether the Service has been assigned a
LoadBalancerIP address (or hostname, which the controller resolves in the background on providers such as AWS that publish a hostname rather than an IP). A Gateway with no external address cannot be healthy. -
Istio proxy configuration sync (optional): when the Helm value
enableIstioProxyStatusCheckis set totrue, and the Service matches an Istio ingress gateway (selectoristio: ingressgateway), the controller additionally asksistiodwhether the Gateway'sistio-proxyhas received all of its configuration. This is the same sync information reported byistioctl proxy-status. All of Listeners, Routes, Clusters and Endpoints (LDS/RDS/CDS/EDS) must report asSYNCED. This catches cases where the Gateway Pod isReadybut has not yet received its Istio configuration, and so cannot correctly route traffic. This check is disabled by default; enable it if you want GSLB failover to also account for Istio configuration propagation delays.The controller checks the first Pod that matches the Service's selector. The Istio proxy check is treated as not yet healthy if the Service has no selector, no Pod matches it, the Pod has no
istio-proxycontainer,istiodcannot be reached, or any of the four configuration types is not synced.
A Gateway must pass all of the checks that are enabled for it to be marked healthy. When a Gateway is not healthy, the controller publishes none in place of its address (see below), so that no traffic is directed to it.
When a Gateway Becomes Healthy or Unhealthy
| Check | Evaluated | Becomes unhealthy | Becomes healthy |
|---|---|---|---|
| Endpoint readiness | Every reconcile, and whenever the Service's EndpointSlices change | Immediately, when no endpoint is Ready | As soon as at least one endpoint is Ready |
| External address | Every reconcile | Immediately, when no address is assigned | As soon as an address is assigned |
| Istio proxy sync (optional) | Until the Gateway is first marked healthy | Not re-evaluated once the Gateway has been marked healthy | After three consecutive checks find the proxy fully synced |
For the endpoint and address checks, there is no "wait and see" grace period for either transition.
The Istio proxy sync check is a gate on initial readiness. While a Gateway has not yet passed it, the controller re-checks every 5 seconds, so a Gateway whose configuration is stable becomes healthy after roughly 10 to 15 seconds. Any check that finds the proxy not fully synced resets the count to zero. This avoids marking a Gateway healthy while its proxy configuration is still converging.
Once a Gateway has passed the Istio proxy sync check, the controller remembers the result and does not run the check again until the controller restarts. If the Gateway later loses its Istio configuration while its Pods remain Ready, GSLB does not detect that condition. Ongoing failure detection relies on endpoint readiness and the external address.
Propagating Health Between Clusters and Regions
Each cluster's Tetrate GSLB controller only has direct visibility into the Gateways running in its own cluster. To build a complete picture across all clusters and regions, every controller publishes two kinds of TXT record, and reads the equivalent records written by every other cluster:
- Gateway address record: for each annotated Gateway, a TXT record containing the Gateway's external address when the Gateway is healthy, or
nonewhen it is not. Other clusters read these records to decide which addresses to include in the Regional Gateway and Failover Gateway A records. - Heartbeat record (
health-001-<name>.<dns-zone>): a timestamp, refreshed by each controller, showing that the controller (and itsexternal-dns) in that cluster or region is alive and able to update DNS. Other clusters treat a stale heartbeat as an unhealthy cluster or region.
The heartbeat is published in two zones:
- Local heartbeat: written to the Local Private DNS Zone, and read by other clusters in the same region, to build the Regional Gateway A record.
- Global heartbeat: written to the Global Private DNS Zone, and read by other regions, to build the Failover Gateway A record.
A heartbeat is considered stale — and the corresponding cluster or region is treated as unhealthy — if its timestamp has not been refreshed within the configured healthy duration:
localTXTRecordHealthyDurationSeconds(default: 30 seconds) for local (regional) heartbeatsglobalTXTRecordHealthyDurationSeconds(default: 30 seconds) for global heartbeats
Heartbeats are refreshed roughly every reconcileRequeueSeconds (default: 10 seconds; see Metrics for counters that track these updates), giving a healthy margin under the default 30-second staleness window.
If a Gateway or an entire cluster becomes unhealthy, its address is removed from the Regional Gateway and Failover Gateway DNS records. However, if removing unhealthy addresses would leave a DNS record with no healthy targets at all, the Tetrate GSLB solution deliberately leaves the existing (stale) record in place rather than publishing an empty answer, since DNS providers handle empty record sets inconsistently. In this situation, clients continue to be directed towards the last-known addresses until at least one Gateway recovers.
Self-Healing and Rate-Limit Protection
DNS providers can rate-limit or throttle frequent writes, which could otherwise cause a cluster's own heartbeat to appear stale (and so falsely trigger a failover) simply because the controller was unable to write its update in time. To protect against this, if the controller detects that its own heartbeat is stale, it writes a self-heal heartbeat timestamped up to selfHealInterval (default: 3 minutes) into the future. This suppresses further, unnecessary failover flapping caused purely by write throttling, while still allowing a genuine outage to be detected once the self-healed timestamp naturally expires.
You can monitor this behavior using the tetrate_gslb_controller_local_healthcheck_is_recovering_from_rate_limiting and tetrate_gslb_controller_global_healthcheck_is_recovering_from_rate_limiting metrics, and observe overall health and failover status using the tetrate_gslb_controller_cluster_healthy, tetrate_gslb_controller_region_healthy and tetrate_gslb_controller_failover_in_progress metrics — see Metrics for details.