The check compared Newbook's FIRST tariff (typically flexi B&B) against
Booking.com's lead-in card (often prepaid) — apples vs oranges, e.g.
21 Jul flagged B&B FLEX £289 vs a prepaid BC rate whose true comparable
was B&B PPAY £279.
- classify Newbook tariffs from names/descriptions (PPAY/prepay/advance/
ADV/NRF/saver = prepaid; DBB/dinner/half board = dinner-inclusive)
- use the scraped BC rate's flags (free_cancellation/no_prepayment/
breakfast_included) to pick the cheapest COMPARABLE tariff per date,
excluding tariffs not bookable for that date (success=false, min-stay
>1, unmet advance-purchase windows)
- tiered fallback (matched -> any non-dinner -> any -> legacy headline),
recorded per alert in room_category as 'TARIFF vs BC basis'
- shared gather_comparisons() now drives both the daily job and
GET /competitors/parity (issues gain newbook_tariff, booking_basis,
match_quality, expected_rate)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- config: parity_markup_value/_unit + parity_tolerance_value/_unit
(pct|gbp), legacy *_pct keys still read as fallback
- expected rate = newbook + £X or newbook × (1 + X%); breach test uses
the tolerance in its own unit
- Settings parity tab: unit selects + live worked example line
- Parity Alerts tab: unit-aware description, badge now shows £ deviation
alongside %
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parity was read-only — the alerts table had no producer, so the Market
View badge could never fire. Now:
- jobs/check_rate_parity.py: daily 06:45 job comparing own Booking.com
lead-in rate vs cheapest Newbook rate per date, measured against an
EXPECTED markup (we deliberately price Booking.com higher to cover
commission): alert when deviation from newbook*(1+markup%) exceeds the
tolerance. Creates/updates active alerts, auto-resolves dates back in
line, leaves acknowledged dates alone.
- config keys: parity_check_enabled, parity_expected_markup_pct,
parity_tolerance_pct (system_config)
- POST /competitors/parity/check manual trigger; GET /parity now uses the
same markup/tolerance and cheapest-across-categories Newbook rate
- Settings -> Rate Parity tab: markup %, tolerance %, enable toggle,
run-now with result summary
- Market View -> Parity Alerts tab: status-filtered list w/ acknowledge
- Market View -> Hotels: direct-link dropdown per competitor (new PUT
/competitors/hotels/{id}/direct-link) — closes the never-written
direct_hotel_id gap so the matrix direct-rates sub-row can populate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Direct-rates sub-row in Market View was dead: the frontend filters
competitors on direct_hotel_id but /matrix and /hotels never returned it.
Add direct_hotel_id to both queries (+ HotelResponse), and only render the
"Direct" sub-row when a hotel actually has a non-null direct rate (was
rendering all-dashes on an empty {} object).
Remove the pause-on-block flow entirely — dormant since rotate-on-block
replaced it (nothing set booking_scraper_paused=true after set_scraper_paused
was dropped): is_scraper_paused, /config/unpause, the /scrape paused guard,
ScraperStatusResponse.paused/pause_until, and the frontend Paused badge +
Unpause button. Trim now-unused datetime import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Proxy config, DataImpulse sticky-session username syntax, and Playwright/
httpx proxy builders now live in one place (services/proxy.py) instead of
being duplicated across the Booking.com backend and the /config/proxy test
endpoint. Both scrapers consume it.
- services/proxy.py: load_config/normalize (DB-authoritative, env fallback),
new_session_id, username, playwright_proxy, httpx_proxy_url
- PlaywrightLocalBackend delegates proxy building to the module
- get_scraper_backend factory uses proxy.load_config (one resolution path)
- test_proxy_config endpoint uses the shared URL builder; httpx proxies= ->
proxy= (forward-compatible, 0.28-safe)
- Direct booking-engine scraper (httpx) can now route through the same proxy,
gated by the direct_scraper_use_proxy flag (default off, plumbing ready)
Dead code removed: set_scraper_paused (never called — rotate-on-block
replaced pause-on-block), get_competitor_matrix / get_hotels_list /
update_hotel_tier (endpoints have their own SQL), unused PROXY_KEYS tuple.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The location_search_url column existed but was dead — the scraper always
rebuilt the URL from the free-text location name. Now the Location
Configuration form takes a "Booking.com search URL" field: paste the
address-bar URL from a real search and the scraper lifts ss/dest_id/
dest_type from it (the most reliable destination pin). Falls back to
dest_id, then plain name. Server also extracts dest_id from the URL for
the column and derives a display name from ss when none is typed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Free-text ss= destination resolution is non-deterministic: tonight's
30-day run resolved "Stow on the Wold" to St. Wolfgang, Austria for 8 of
30 dates, saving Salzkammergut hotel rates into the matrix. dest_id +
dest_type=city in the search URL pins the destination.
- booking_scrape_config gains a dest_id column (idempotent ALTER)
- scrape_location_search/_build_search_url thread dest_id through
- /config/location accepts dest_id
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "Scraper Proxy" tab: enable toggle, host/port/username/password/
country, Save, and a Test Connection button that reports the live exit
IP + country through the proxy
- Backend proxy config now lives in system_config (DB authoritative when
booking_proxy_enabled is set; BOOKING_PROXY_* env vars are the fallback)
- Dedicated /config/proxy GET/POST/test endpoints; password is write-only
(never returned, blank keeps the stored value) and masked in /config/system
- Surface proxy status keys in the read-only System tab
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- matrix response gains last_scraped per date (max scraped_at across all
hotels), so column headers show the newest scrape touching the date even
when the visible hotels were on a failed page
- cells >1h older than the column's latest scrape render italic with *
- every cell tooltip now includes the datestamp the price was scraped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
