- Route through DataImpulse residential proxy when BOOKING_PROXY_* env
vars are set (empty = direct connection, unchanged behaviour)
- Sticky one IP per session via DataImpulse sessid; rotate_session()
burns the IP for a fresh one
- Rotate-on-block: retry a date on a new IP when the page is challenged
or page 1 renders zero hotels (up to 3 IPs; single attempt sans proxy)
- Persistent, pre-warmed context across dates so cache stays hot
(~5 MB first page, ~0.3 MB per date after) instead of per-date cold loads
- Block images/media/fonts and third-party ad/consent/analytics hosts to
cut bandwidth; never touch the anti-bot challenge (awswaf)
- Faster inter-page pacing now that a burned IP is cheap
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page-load timeout was logged and skipped, so a scrape could 'succeed'
with only page 1 of results — and the not_listed flagging then marked
every page-2 hotel absent, suppressing their last known rates.
- ScraperResult now tracks pages_requested/pages_ok
- scrape_date skips not_listed flagging when pages failed or when the
scrape saw <60% of the date's 7-day coverage baseline; scraped rates
are still saved, unseen hotels keep last known rate + scrape time
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Location-search results aren't a fixed hotel set — a sold-out hotel drops
out and its last 'available' rate would remain the latest row for that
date, reading as a live price and skewing market averages. On each
successful per-date scrape, insert a NULL-rate 'not_listed' row for every
active hotel missing from the results (skipped if the parse found nothing,
which indicates scraper fault not absence). Failed/blocked scrapes write
nothing, so genuinely-stale data remains distinguishable by scraped_at.
Also: fix the DOW analysis to pick latest-then-filter so a not_listed
latest row drops the date instead of resurfacing an older rate, and widen
booking_com_id to VARCHAR(255) (some Booking slugs exceed 50 chars).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scraped room descriptions exceed VARCHAR(100) (StringDataRightTruncation),
and the failed insert poisoned the transaction so every subsequent save in
the batch died with InFailedSqlTransaction — scrapes reported success with
0 rows saved. Widen the column (with ALTER for existing tables) and roll
back after a failed save.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bookability showed no availability because newbook_occupancy_report_data
was never populated: add reports_occupancy client method and
sync_occupancy job, run before rates in both Sync Now and the daily
schedule (single fast API call)
- Category order: default display_order to the Newbook category id on
sync (was 0 → alphabetical), preserve manual order on re-sync, extend
PATCH to accept display_order, add up/down reorder arrows in Settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rates Newbook instance returns categories as category_id/category_name
on each site; also skip inactive categories.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- require_cap was a Depends-factory but every call site uses it inline;
make it an inline checker (fixes 500 on /analysis/hotels, /direct/*)
- /analysis/comparison returned a per-date matrix the frontend never read;
return per-hotel aggregates (our/their avg, price index) and default to
all active competitors so the Market Comparison table works without params
- Room categories were never populated (lost in port): add sites_list fetch
to the Newbook client, categories list/sync/toggle endpoints, and a
Settings card — without included categories every rates sync exits early
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
