Add stale-scrape health alerting (email + in-app banner)
Silent scraper breakage (200-with-empty-body) went unnoticed for 16 days because nothing flagged it. Now, after each daily direct scrape, a health check compares every enabled competitor's freshest captured rate against a staleness threshold (default 5 days, config: scrape_health_stale_days). State lives in a new direct_scrape_health table so we alert on transitions only — one email when a competitor goes stale, one when it recovers, never a daily repeat. Email uses the stack's shared SMTP integration via central_settings (recipient: scrape_health_alert_email, else SMTP reply_to/ from) — no new mail secrets in this app. The current stale set also drives an in-app warning banner on the Direct Rates page (GET /direct/health), with an on-demand re-check endpoint (POST /direct/health/check). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b916f1a09b
commit
ea240b3f96
6 changed files with 261 additions and 1 deletions
|
|
@ -331,6 +331,19 @@ CREATE INDEX IF NOT EXISTS idx_direct_rates_scraped ON direct_rates(scraped_a
|
|||
ALTER TABLE booking_com_hotels ADD COLUMN IF NOT EXISTS direct_hotel_id INTEGER
|
||||
REFERENCES direct_competitor_hotels(id) ON DELETE SET NULL;
|
||||
|
||||
-- ============================================
|
||||
-- DIRECT SCRAPE HEALTH (per-hotel freshness state, drives stale-data alerting)
|
||||
-- ============================================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS direct_scrape_health (
|
||||
hotel_id INTEGER PRIMARY KEY REFERENCES direct_competitor_hotels(id) ON DELETE CASCADE,
|
||||
state TEXT NOT NULL DEFAULT 'ok', -- 'ok' | 'stale'
|
||||
fresh_as_of TIMESTAMPTZ, -- freshest direct_rates.scraped_at seen for this hotel
|
||||
stale_since TIMESTAMPTZ, -- when it first went stale (cleared on recovery)
|
||||
notified_at TIMESTAMPTZ, -- last time a transition email was sent
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Pin the Booking.com destination: free-text ss= searches non-deterministically
|
||||
-- resolve to the wrong place (Stow on the Wold once matched St. Wolfgang, AT).
|
||||
ALTER TABLE booking_scrape_config ADD COLUMN IF NOT EXISTS dest_id VARCHAR(32);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue