rates/backend/services/scraper_backends
jtricerolph 349c795708 Scraper stealth: headful+xvfb, playwright-stealth, coherent UA, warm-up, click-paginate
Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
  WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
  platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
  Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
  consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
  URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 16:26:01 +00:00
..
__init__.py Add Rate Monitor app — Booking.com + direct booking engine competitor rates 2026-07-05 12:06:30 +00:00
base.py Scraper: don't flag not_listed from partial scrapes 2026-07-05 15:29:06 +00:00
playwright_local.py Scraper stealth: headful+xvfb, playwright-stealth, coherent UA, warm-up, click-paginate 2026-07-05 16:26:01 +00:00