Scraper stealth: headful+xvfb, playwright-stealth, coherent UA, warm-up, click-paginate
Tier 1+2 anti-detection to stop Booking.com throttling page 2: - Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader WebGL, empty plugins, missing chrome.runtime - playwright-stealth patches navigator.webdriver/plugins/WebGL vendor - Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints + platform (dropped the Firefox/Safari UA strings — a mismatched UA on a Chromium engine is a stronger tell than no rotation) - Homepage warm-up so search requests carry real session cookies + cookie consent dismiss - Paginate by clicking next (offset= deep-link was the page-2 tell), offset URL kept as fallback - Viewport jitter, mouse movement, longer scroll, selector retry on lazy load Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
b712196262
commit
349c795708
3 changed files with 148 additions and 36 deletions
|
|
@ -7,6 +7,7 @@ RUN apt-get update && apt-get install -y \
|
|||
libpq-dev \
|
||||
postgresql-client \
|
||||
curl \
|
||||
xvfb \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY requirements.txt .
|
||||
|
|
@ -19,4 +20,7 @@ COPY . .
|
|||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
# Run under a virtual X display so Chromium launches headful (headful fingerprints
|
||||
# far cleaner than headless — real GPU strings, plugins, chrome.runtime).
|
||||
CMD ["xvfb-run", "-a", "--server-args=-screen 0 1920x1080x24", \
|
||||
"uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue