Scraper stealth: headful+xvfb, playwright-stealth, coherent UA, warm-up, click-paginate

Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
  WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
  platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
  Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
  consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
  URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-05 16:26:01 +00:00
parent b712196262
commit 349c795708
3 changed files with 148 additions and 36 deletions

View file

@ -7,6 +7,7 @@ RUN apt-get update && apt-get install -y \
libpq-dev \
postgresql-client \
curl \
xvfb \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
@ -19,4 +20,7 @@ COPY . .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
# Run under a virtual X display so Chromium launches headful (headful fingerprints
# far cleaner than headless — real GPU strings, plugins, chrome.runtime).
CMD ["xvfb-run", "-a", "--server-args=-screen 0 1920x1080x24", \
"uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]