False-positive health alerts on data-sufficiency probes

Resolved

A transient Neon pooler blip triggered 31 false-positive red alerts on our internal probe runner. No user-facing impact; alerts were retracted within 90 minutes.

Started: 20 Apr 2026, 21:10 UTC
Resolved: 20 Apr 2026, 22:45 UTC
Severity: minor

Affected products

GeraClinic
GeraHome
GeraEats
API Gateway

Timeline

Investigating
20 Apr 2026, 21:15 UTC
Internal probe runner reporting elevated failures across multiple products. Investigating whether this is a shared infrastructure issue or a probe bug.
Identified
20 Apr 2026, 21:42 UTC
Traced to a single Neon pooler blip at 21:10 UTC. All user-facing requests succeeded. Probe runner lacked retry logic, which amplified a single blip into many alerts.
Resolved
20 Apr 2026, 22:45 UTC
Added retry-with-backoff (1s/3s/9s) to probe runner. A 2-consecutive-failure rule now gates alerts. Closed.