Add stale-scrape health alerting (email + in-app banner)

Silent scraper breakage (200-with-empty-body) went unnoticed for 16 days
because nothing flagged it. Now, after each daily direct scrape, a health
check compares every enabled competitor's freshest captured rate against a
staleness threshold (default 5 days, config: scrape_health_stale_days).

State lives in a new direct_scrape_health table so we alert on transitions
only — one email when a competitor goes stale, one when it recovers, never a
daily repeat. Email uses the stack's shared SMTP integration via
central_settings (recipient: scrape_health_alert_email, else SMTP reply_to/
from) — no new mail secrets in this app. The current stale set also drives an
in-app warning banner on the Direct Rates page (GET /direct/health), with an
on-demand re-check endpoint (POST /direct/health/check).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-08-13 15:44:03 +00:00
parent b916f1a09b
commit ea240b3f96
6 changed files with 261 additions and 1 deletions

View file

@ -4,7 +4,7 @@ import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query'
import Plot from 'react-plotly.js'
import {
Building2, RefreshCw, Plus, ChevronDown, ChevronRight, ChevronUp,
Clock, LineChart, Settings2, X,
Clock, LineChart, Settings2, X, AlertTriangle,
} from 'lucide-react'
import api from '../api'
import { useAuth } from '../components/AuthGate'
@ -101,6 +101,13 @@ interface HistoryTarget {
title: string
}
interface StaleHotel {
hotel_id: number
name: string
fresh_as_of: string | null
stale_since: string | null
}
const DAYS = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
const fmt = (v: number | null | undefined) => v != null ? `£${Number(v).toFixed(2)}` : '—'
const age = (ts: string | null) => {
@ -158,6 +165,12 @@ export default function DirectRates() {
queryFn: () => api.get('/direct/hotels').then(r => r.data),
})
const { data: health } = useQuery<{ stale: StaleHotel[] }>({
queryKey: ['direct-health'],
queryFn: () => api.get('/direct/health').then(r => r.data),
refetchInterval: 5 * 60 * 1000,
})
const { data: dates, isLoading: datesLoading } = useQuery<DatesResponse>({
queryKey: ['direct-dates', selectedHotel, fromDate, toDate],
queryFn: () => api.get(`/direct/hotels/${selectedHotel}/dates`, {
@ -198,6 +211,23 @@ export default function DirectRates() {
</div>
</div>
{health && health.stale.length > 0 && (
<div style={{
display: 'flex', gap: 10, alignItems: 'flex-start', padding: '12px 16px', marginBottom: 16,
background: '#fef3c7', border: '1px solid #fcd34d', borderRadius: 8, color: '#92400e', fontSize: 13,
}}>
<AlertTriangle size={18} strokeWidth={1.75} style={{ flexShrink: 0, marginTop: 1 }} />
<div>
<strong>Stale competitor data.</strong>{' '}
{health.stale.map(s => s.name).join(', ')} {health.stale.length === 1 ? 'has' : 'have'} had
no fresh scrape data for several days the scraper may be broken.
<div style={{ marginTop: 3, fontSize: 12, opacity: 0.85 }}>
{health.stale.map(s => `${s.name}: last data ${s.fresh_as_of ? age(s.fresh_as_of) : 'never'}`).join(' · ')}
</div>
</div>
</div>
)}
<div className="sub-nav">
{TABS.map(t => (
<button key={t} className={`sub-nav-item${tab === t ? ' active' : ''}`}