Parity logic:
- Extract _fetch_latest_alerts() and _upsert_alerts() helpers so the alert
upsert loop is no longer duplicated between the daily job and per-date runs
- Add run_parity_check_for_date(date) which runs the full comparison +
alert upsert for a single date
Scraper integration:
- _safe_parity_recheck(date) wrapper (never raises) called after each
successful date in both search-results and hotel-page workers; hotel-page
mode waits until all hotels for the date are done before rechecking
API:
- POST /competitors/parity/check-date?rate_date=YYYY-MM-DD for manual recheck
Frontend:
- Recheck button on every parity alert row (all statuses); invalidates
alerts list and badge count on success
- parity-alert-count badge polls every 15s (was 60s) so new alerts from
the scheduled job or post-scrape rechecks appear quickly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend: POST /competitors/scrape/scheduled triggers run_scheduled_booking_scrape
in a background task — repopulates the queue with today's high/medium/low
priority dates and processes it. Returns 400 if scraper is disabled, 409 if
already running.
Frontend: button in the Automatic Schedule card to trigger the scheduled job
manually (useful after an interrupted overnight run). Manual Scrape description
now shows which backend mode is active (hotel-page vs search-results) so it's
clear what the date range scrape actually does.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Original scrapy approach: extract <li> condition lines from each rate plan row
and text-search for known keywords. No match = null (unknown), not assumed false.
Adds breakfast_text, cancel_text, payment_text columns to booking_com_rates.
Booleans now nullable (null = not mentioned, true/false = explicit signal).
JS extraction searches all <li> items in the row (falls back to newline-split
innerText if none). Breakfast: 'breakfast' keyword. Cancel: 'free cancellation',
'non-refundable', 'total cost to cancel', 'fully chargeable'. Payment: 'no
prepayment', 'pay at the property', 'pay online'. data-fltrs used as fallback.
API snapshot endpoint now returns breakfast/cancel/payment text strings. Old
boolean-only rows degrade gracefully to derived labels.
Modal plan rows replace the fixed meal|cancel|price column layout with a single
stacked conditions cell: 0-3 lines depending on what the page actually shows.
Breakfast green when 'included', cancel green when 'Free cancellation…'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A new toolbar row on the Rate Matrix lets users pick a reference datetime
(defaults to 24h ago, with 24h/3d/7d presets) and see ▲/▼ triangles
next to each competitor's BAR where the rate has moved ≥50p since then.
Two backend endpoints:
- GET /competitors/rate-changes?since= — static datetime comparison
- GET /competitors/rate-changes-vs-own — dynamic: diffs against the
timestamp our own Newbook rate last changed per date (highlights
competitor moves made in response to our own pricing decisions)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse username contains semicolons (;sessid.ID) that weren't being
URL-encoded in httpx_proxy_url, causing silent parse failures. Also use
httpx.Proxy object instead of raw string (consistent with httpx_proxy()),
and capture repr(e) so empty-message exceptions show their type.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Header-based auth made no difference; the real issue is the hotel network
firewall blocking HTTPS CONNECT tunnels on port 823. Reverted to URL-embedded
credentials (original approach). Fix requires DataImpulse to enable port 443.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Embedding credentials in the proxy URL (http://user:pass@host:port) was
breaking HTTPS CONNECT tunnels on the hotel network. Switching to separate
username/password fields (Playwright) and httpx.Proxy(auth=...) sends a
Proxy-Authorization header instead, which passes through correctly.
With DataImpulse IP whitelisting the 407 round-trip is skipped anyway so
there is no latency penalty.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old Stocks block_id has 0 in position 2 (not the persons count) so all
its rates were stored as max_persons=0 and deprioritised in the matrix.
Now reads max_persons from the visible 'Max persons: N' span (matching
the original scrapy spider), with aria-label/title fallback. block_id
segment 2 is unreliable across different hotel layouts.
Also treat max_persons=0 as unknown in the matrix ORDER BY priority.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Include tier=own in get_active_hotels() so the own hotel Booking.com
listing is scraped alongside competitors. Change matrix max_persons
filter from hard WHERE to ORDER BY priority so hotels with only
1-person rates (e.g. Old Stocks) still show up rather than being
silently excluded.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: restrict hotel-page scraper to 'competitor' tier only (was own+competitor); own hotel rates come from Newbook API, not Booking.com
- Backend: POST /competitors/discover — runs search-results scrape for one date to find market hotels regardless of current backend setting
- Backend: POST /competitors/hotels — add hotel manually from Booking.com URL + name + tier; upserts on slug conflict
- Frontend (Settings tab): Discover Market Hotels button added below manual date-range scrape
- Frontend (Hotels tab): Add Hotel form at top — paste URL (auto-derives name from slug), choose tier, submit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Modal now shows two sections:
- Current availability: all room types with rooms-left count, then each rate
plan (meal plan × cancel policy × price) from the latest scrape batch
- Rate history chart: one line per room type, cheapest 2-adult rate per
scrape run (max_persons filter added to exclude 1-adult variants)
New endpoint: GET /competitors/hotels/{id}/rate-snapshot/{date}
Returns all rate plan variants grouped by room type from the latest batch.
Handles legacy single-row data (pre hotel-page scraper) gracefully.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switches booking.com scraping from search-results page (Cloudflare-targeted)
to individual hotel property pages (not CF-protected). Each page load returns
all room types, all rate plan variants (room-only/B&B × refundable/non-ref ×
1-2 adults), and availability counts.
Key changes:
- New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF
- booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(),
_scrape_hotels_concurrent() — sharded by hotel so one proxy session covers
all dates for one hotel (looks human)
- schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates
- competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker
so DISTINCT ON returns cheapest 2-adult rate from latest batch
Enable via Settings → Scraper Backend → playwright_hotel_page
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two rapid POST /scrape requests could both pass the lock check before
either background task acquired SCRAPE_LOCK, queuing two sequential
scrapes for the same date range. The second would hit Cloudflare after
the first already succeeded, causing repeated retries.
_SCRAPE_PENDING is set on submission and cleared when the task starts,
closing the gap between the 409 check and lock acquisition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: threading.Lock held indefinitely when Playwright browser
hangs inside run_in_executor (finally never fires from the async side).
Fixes:
- _acquire_scrape_lock/_release_scrape_lock track monotonic timestamp
- POST /competitors/scrape/reset force-releases the lock and marks any
running batch as interrupted (queue rows stay intact for retry)
- GET /competitors/status now includes lock_held_seconds
- APScheduler watchdog job every 30 min auto-releases if held >3h
- Settings → Scraper Proxy tab shows live lock status (green/amber)
with a Force Reset button requiring confirmation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sold-out cells: amber (was red), show last known available rate with
strikethrough. Past dates: grey, same strikethrough treatment, no eye
icon (Booking.com won't serve past availability), history chart still
accessible. Date column headers also hide the eye link for past dates.
Backend adds a second query returning last_available_rate (most recent
available + non-null rate_gross) per hotel+date for the full range.
Icons are now 12px and stacked below the rate text in a flex-column
cell layout. Price index badge sits between rate and icons.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace full-cell link with two small icon buttons in each matrix cell:
- Eye icon links to the hotel's Booking.com page for that check-in date
- LineChart icon opens a history modal (Plotly multi-line chart)
The history chart plots best available rate over scrape runs, with one
line per room type so that changes in which room is cheapest show as
separate traces rather than a single jumpy line.
Backend: GET /competitors/hotels/{id}/rate-history/{stay_date} groups
by scrape batch, takes MIN(rate_gross) per room type per run, returns
series [{room_type, points: [{t, rate}]}].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New GET /competitors/own-direct-rates: cheapest bookable non-dinner
Newbook tariff per date (same selection rules as the parity check).
'Show direct rates' now renders a 'Direct (Newbook)' sub-row under the
own-hotel row alongside the competitors' scraped direct rows; cell
tooltip names the tariff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The check compared Newbook's FIRST tariff (typically flexi B&B) against
Booking.com's lead-in card (often prepaid) — apples vs oranges, e.g.
21 Jul flagged B&B FLEX £289 vs a prepaid BC rate whose true comparable
was B&B PPAY £279.
- classify Newbook tariffs from names/descriptions (PPAY/prepay/advance/
ADV/NRF/saver = prepaid; DBB/dinner/half board = dinner-inclusive)
- use the scraped BC rate's flags (free_cancellation/no_prepayment/
breakfast_included) to pick the cheapest COMPARABLE tariff per date,
excluding tariffs not bookable for that date (success=false, min-stay
>1, unmet advance-purchase windows)
- tiered fallback (matched -> any non-dinner -> any -> legacy headline),
recorded per alert in room_category as 'TARIFF vs BC basis'
- shared gather_comparisons() now drives both the daily job and
GET /competitors/parity (issues gain newbook_tariff, booking_basis,
match_quality, expected_rate)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- config: parity_markup_value/_unit + parity_tolerance_value/_unit
(pct|gbp), legacy *_pct keys still read as fallback
- expected rate = newbook + £X or newbook × (1 + X%); breach test uses
the tolerance in its own unit
- Settings parity tab: unit selects + live worked example line
- Parity Alerts tab: unit-aware description, badge now shows £ deviation
alongside %
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parity was read-only — the alerts table had no producer, so the Market
View badge could never fire. Now:
- jobs/check_rate_parity.py: daily 06:45 job comparing own Booking.com
lead-in rate vs cheapest Newbook rate per date, measured against an
EXPECTED markup (we deliberately price Booking.com higher to cover
commission): alert when deviation from newbook*(1+markup%) exceeds the
tolerance. Creates/updates active alerts, auto-resolves dates back in
line, leaves acknowledged dates alone.
- config keys: parity_check_enabled, parity_expected_markup_pct,
parity_tolerance_pct (system_config)
- POST /competitors/parity/check manual trigger; GET /parity now uses the
same markup/tolerance and cheapest-across-categories Newbook rate
- Settings -> Rate Parity tab: markup %, tolerance %, enable toggle,
run-now with result summary
- Market View -> Parity Alerts tab: status-filtered list w/ acknowledge
- Market View -> Hotels: direct-link dropdown per competitor (new PUT
/competitors/hotels/{id}/direct-link) — closes the never-written
direct_hotel_id gap so the matrix direct-rates sub-row can populate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Direct-rates sub-row in Market View was dead: the frontend filters
competitors on direct_hotel_id but /matrix and /hotels never returned it.
Add direct_hotel_id to both queries (+ HotelResponse), and only render the
"Direct" sub-row when a hotel actually has a non-null direct rate (was
rendering all-dashes on an empty {} object).
Remove the pause-on-block flow entirely — dormant since rotate-on-block
replaced it (nothing set booking_scraper_paused=true after set_scraper_paused
was dropped): is_scraper_paused, /config/unpause, the /scrape paused guard,
ScraperStatusResponse.paused/pause_until, and the frontend Paused badge +
Unpause button. Trim now-unused datetime import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Proxy config, DataImpulse sticky-session username syntax, and Playwright/
httpx proxy builders now live in one place (services/proxy.py) instead of
being duplicated across the Booking.com backend and the /config/proxy test
endpoint. Both scrapers consume it.
- services/proxy.py: load_config/normalize (DB-authoritative, env fallback),
new_session_id, username, playwright_proxy, httpx_proxy_url
- PlaywrightLocalBackend delegates proxy building to the module
- get_scraper_backend factory uses proxy.load_config (one resolution path)
- test_proxy_config endpoint uses the shared URL builder; httpx proxies= ->
proxy= (forward-compatible, 0.28-safe)
- Direct booking-engine scraper (httpx) can now route through the same proxy,
gated by the direct_scraper_use_proxy flag (default off, plumbing ready)
Dead code removed: set_scraper_paused (never called — rotate-on-block
replaced pause-on-block), get_competitor_matrix / get_hotels_list /
update_hotel_tier (endpoints have their own SQL), unused PROXY_KEYS tuple.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The location_search_url column existed but was dead — the scraper always
rebuilt the URL from the free-text location name. Now the Location
Configuration form takes a "Booking.com search URL" field: paste the
address-bar URL from a real search and the scraper lifts ss/dest_id/
dest_type from it (the most reliable destination pin). Falls back to
dest_id, then plain name. Server also extracts dest_id from the URL for
the column and derives a display name from ss when none is typed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Free-text ss= destination resolution is non-deterministic: tonight's
30-day run resolved "Stow on the Wold" to St. Wolfgang, Austria for 8 of
30 dates, saving Salzkammergut hotel rates into the matrix. dest_id +
dest_type=city in the search URL pins the destination.
- booking_scrape_config gains a dest_id column (idempotent ALTER)
- scrape_location_search/_build_search_url thread dest_id through
- /config/location accepts dest_id
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "Scraper Proxy" tab: enable toggle, host/port/username/password/
country, Save, and a Test Connection button that reports the live exit
IP + country through the proxy
- Backend proxy config now lives in system_config (DB authoritative when
booking_proxy_enabled is set; BOOKING_PROXY_* env vars are the fallback)
- Dedicated /config/proxy GET/POST/test endpoints; password is write-only
(never returned, blank keeps the stored value) and masked in /config/system
- Surface proxy status keys in the read-only System tab
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- matrix response gains last_scraped per date (max scraped_at across all
hotels), so column headers show the newest scrape touching the date even
when the visible hotels were on a failed page
- cells >1h older than the column's latest scrape render italic with *
- every cell tooltip now includes the datestamp the price was scraped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
