Proxy config, DataImpulse sticky-session username syntax, and Playwright/
httpx proxy builders now live in one place (services/proxy.py) instead of
being duplicated across the Booking.com backend and the /config/proxy test
endpoint. Both scrapers consume it.
- services/proxy.py: load_config/normalize (DB-authoritative, env fallback),
new_session_id, username, playwright_proxy, httpx_proxy_url
- PlaywrightLocalBackend delegates proxy building to the module
- get_scraper_backend factory uses proxy.load_config (one resolution path)
- test_proxy_config endpoint uses the shared URL builder; httpx proxies= ->
proxy= (forward-compatible, 0.28-safe)
- Direct booking-engine scraper (httpx) can now route through the same proxy,
gated by the direct_scraper_use_proxy flag (default off, plumbing ready)
Dead code removed: set_scraper_paused (never called — rotate-on-block
replaced pause-on-block), get_competitor_matrix / get_hotels_list /
update_hotel_tier (endpoints have their own SQL), unused PROXY_KEYS tuple.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The location_search_url column existed but was dead — the scraper always
rebuilt the URL from the free-text location name. Now the Location
Configuration form takes a "Booking.com search URL" field: paste the
address-bar URL from a real search and the scraper lifts ss/dest_id/
dest_type from it (the most reliable destination pin). Falls back to
dest_id, then plain name. Server also extracts dest_id from the URL for
the column and derives a display name from ss when none is typed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Free-text ss= destination resolution is non-deterministic: tonight's
30-day run resolved "Stow on the Wold" to St. Wolfgang, Austria for 8 of
30 dates, saving Salzkammergut hotel rates into the matrix. dest_id +
dest_type=city in the search URL pins the destination.
- booking_scrape_config gains a dest_id column (idempotent ALTER)
- scrape_location_search/_build_search_url thread dest_id through
- /config/location accepts dest_id
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Route through DataImpulse residential proxy when BOOKING_PROXY_* env
vars are set (empty = direct connection, unchanged behaviour)
- Sticky one IP per session via DataImpulse sessid; rotate_session()
burns the IP for a fresh one
- Rotate-on-block: retry a date on a new IP when the page is challenged
or page 1 renders zero hotels (up to 3 IPs; single attempt sans proxy)
- Persistent, pre-warmed context across dates so cache stays hot
(~5 MB first page, ~0.3 MB per date after) instead of per-date cold loads
- Block images/media/fonts and third-party ad/consent/analytics hosts to
cut bandwidth; never touch the anti-bot challenge (awswaf)
- Faster inter-page pacing now that a burned IP is cheap
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page-load timeout was logged and skipped, so a scrape could 'succeed'
with only page 1 of results — and the not_listed flagging then marked
every page-2 hotel absent, suppressing their last known rates.
- ScraperResult now tracks pages_requested/pages_ok
- scrape_date skips not_listed flagging when pages failed or when the
scrape saw <60% of the date's 7-day coverage baseline; scraped rates
are still saved, unseen hotels keep last known rate + scrape time
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
