Add stale-scrape health alerting (email + in-app banner)

Silent scraper breakage (200-with-empty-body) went unnoticed for 16 days
because nothing flagged it. Now, after each daily direct scrape, a health
check compares every enabled competitor's freshest captured rate against a
staleness threshold (default 5 days, config: scrape_health_stale_days).

State lives in a new direct_scrape_health table so we alert on transitions
only — one email when a competitor goes stale, one when it recovers, never a
daily repeat. Email uses the stack's shared SMTP integration via
central_settings (recipient: scrape_health_alert_email, else SMTP reply_to/
from) — no new mail secrets in this app. The current stale set also drives an
in-app warning banner on the Direct Rates page (GET /direct/health), with an
on-demand re-check endpoint (POST /direct/health/check).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-08-13 15:44:03 +00:00
parent b916f1a09b
commit ea240b3f96
6 changed files with 261 additions and 1 deletions

View file

@ -41,3 +41,10 @@ def run_scrape_all_direct():
log.error(f"Direct scrape failed for hotel {hotel['id']} ({hotel['name']}): {e}")
log.info("Direct rate scrape complete")
# Flag any competitor whose data has gone stale (and alert on transitions).
try:
from jobs.check_scrape_health import check_scrape_health
check_scrape_health()
except Exception as e:
log.error(f"Scrape health check failed: {e}")