Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page-load timeout was logged and skipped, so a scrape could 'succeed'
with only page 1 of results — and the not_listed flagging then marked
every page-2 hotel absent, suppressing their last known rates.
- ScraperResult now tracks pages_requested/pages_ok
- scrape_date skips not_listed flagging when pages failed or when the
scrape saw <60% of the date's 7-day coverage baseline; scraped rates
are still saved, unseen hotels keep last known rate + scrape time
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
