Silent scraper breakage (200-with-empty-body) went unnoticed for 16 days
because nothing flagged it. Now, after each daily direct scrape, a health
check compares every enabled competitor's freshest captured rate against a
staleness threshold (default 5 days, config: scrape_health_stale_days).
State lives in a new direct_scrape_health table so we alert on transitions
only — one email when a competitor goes stale, one when it recovers, never a
daily repeat. Email uses the stack's shared SMTP integration via
central_settings (recipient: scrape_health_alert_email, else SMTP reply_to/
from) — no new mail secrets in this app. The current stale set also drives an
in-app warning banner on the Direct Rates page (GET /direct/health), with an
on-demand re-check endpoint (POST /direct/health/check).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parity logic:
- Extract _fetch_latest_alerts() and _upsert_alerts() helpers so the alert
upsert loop is no longer duplicated between the daily job and per-date runs
- Add run_parity_check_for_date(date) which runs the full comparison +
alert upsert for a single date
Scraper integration:
- _safe_parity_recheck(date) wrapper (never raises) called after each
successful date in both search-results and hotel-page workers; hotel-page
mode waits until all hotels for the date are done before rechecking
API:
- POST /competitors/parity/check-date?rate_date=YYYY-MM-DD for manual recheck
Frontend:
- Recheck button on every parity alert row (all statuses); invalidates
alerts list and badge count on success
- parity-alert-count badge polls every 15s (was 60s) so new alerts from
the scheduled job or post-scrape rechecks appear quickly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- DISTINCT ON tiebreaker was undefined when scraper writes multiple room
types in one batch (same scraped_at); adding rate_gross ASC ensures we
always pick the cheapest (lead-in / best available) rate, matching like
for like against the Newbook BAR tariff
- Past-date active alerts were never touched (start = today meant they
fell outside the query window); now resolved at the top of each run
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SQLAlchemy's text() parser treated the ':' in '::date' (PostgreSQL cast syntax)
as a second unbound parameter after :start/:end, producing malformed SQL and
crashing the scheduled Booking.com scrape silently with no history entry written.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Medium and low priority tiers were generating full static date ranges
(150 and 185 dates) every day. With a queue limit of 200, high (31) +
medium (150) consumed the entire budget, leaving only ~19 slots for low
priority — causing the observed ~6 month cap.
Medium now selects the 60 oldest-scraped (or never-scraped) dates from
the days 31-180 window; low selects the 30 oldest from days 181-365.
Per-run budget drops from ~365 to ~121 dates, and coverage naturally
cycles through the full year: medium every ~2-3 days, low every ~6-7 days.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema: migrate newbook_occupancy_report_data from single-row upsert to
snapshot model (drop unique constraint, add valid_from / last_verified_at)
matching the pattern used by newbook_current_rates.
Backend: sync_occupancy now inserts a new row only when occupied/available/
maintenance figures change, otherwise bumps last_verified_at. New endpoint
GET /bookability/occupancy-history/{category_id}/{date} returns the timeline.
Rate matrix query updated to DISTINCT ON for the multi-row table.
Frontend: clicking any cell in the Bookability matrix opens a modal with
two stacked Plotly charts — rate history per tariff (step lines, green/red
markers for available/unavailable) and occupancy pick-up over time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch date_last_updated from valid_from (rate change time) to
last_verified_at (last check time). All dates verified in the same daily
run now show a consistent timestamp rather than varying by when rates
last changed. Also commit per-date instead of batching 10 days — prevents
a single API error from rolling back up to 9 preceding committed dates.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
parity_match_mode config ('best_available' default | 'match_terms').
Best available = cheapest bookable non-dinner direct tariff vs the BC
lead-in (already BC's best available) — simplest like-for-like. Term
matching kept as an option for days where BC's cheapest basis differs
from direct's. Settings gains a comparison-basis select.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The check compared Newbook's FIRST tariff (typically flexi B&B) against
Booking.com's lead-in card (often prepaid) — apples vs oranges, e.g.
21 Jul flagged B&B FLEX £289 vs a prepaid BC rate whose true comparable
was B&B PPAY £279.
- classify Newbook tariffs from names/descriptions (PPAY/prepay/advance/
ADV/NRF/saver = prepaid; DBB/dinner/half board = dinner-inclusive)
- use the scraped BC rate's flags (free_cancellation/no_prepayment/
breakfast_included) to pick the cheapest COMPARABLE tariff per date,
excluding tariffs not bookable for that date (success=false, min-stay
>1, unmet advance-purchase windows)
- tiered fallback (matched -> any non-dinner -> any -> legacy headline),
recorded per alert in room_category as 'TARIFF vs BC basis'
- shared gather_comparisons() now drives both the daily job and
GET /competitors/parity (issues gain newbook_tariff, booking_basis,
match_quality, expected_rate)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- config: parity_markup_value/_unit + parity_tolerance_value/_unit
(pct|gbp), legacy *_pct keys still read as fallback
- expected rate = newbook + £X or newbook × (1 + X%); breach test uses
the tolerance in its own unit
- Settings parity tab: unit selects + live worked example line
- Parity Alerts tab: unit-aware description, badge now shows £ deviation
alongside %
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parity was read-only — the alerts table had no producer, so the Market
View badge could never fire. Now:
- jobs/check_rate_parity.py: daily 06:45 job comparing own Booking.com
lead-in rate vs cheapest Newbook rate per date, measured against an
EXPECTED markup (we deliberately price Booking.com higher to cover
commission): alert when deviation from newbook*(1+markup%) exceeds the
tolerance. Creates/updates active alerts, auto-resolves dates back in
line, leaves acknowledged dates alone.
- config keys: parity_check_enabled, parity_expected_markup_pct,
parity_tolerance_pct (system_config)
- POST /competitors/parity/check manual trigger; GET /parity now uses the
same markup/tolerance and cheapest-across-categories Newbook rate
- Settings -> Rate Parity tab: markup %, tolerance %, enable toggle,
run-now with result summary
- Market View -> Parity Alerts tab: status-filtered list w/ acknowledge
- Market View -> Hotels: direct-link dropdown per competitor (new PUT
/competitors/hotels/{id}/direct-link) — closes the never-written
direct_hotel_id gap so the matrix direct-rates sub-row can populate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bookability showed no availability because newbook_occupancy_report_data
was never populated: add reports_occupancy client method and
sync_occupancy job, run before rates in both Sync Now and the daily
schedule (single fast API call)
- Category order: default display_order to the Newbook category id on
sync (was 0 → alphabetical), preserve manual order on re-sync, extend
PATCH to accept display_order, add up/down reorder arrows in Settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- schema defined gross_rate/net_rate but every query (fetch job,
bookability, competitors) uses rate_gross/rate_net — rename the
columns, with an idempotent DO-block migration for existing tables
- roll back the session when a date fails so one bad statement no
longer poisons the whole sync run (every subsequent write was dying
with 'current transaction is aborted')
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
