Add residential proxy support to Booking.com scraper

- Route through DataImpulse residential proxy when BOOKING_PROXY_* env
  vars are set (empty = direct connection, unchanged behaviour)
- Sticky one IP per session via DataImpulse sessid; rotate_session()
  burns the IP for a fresh one
- Rotate-on-block: retry a date on a new IP when the page is challenged
  or page 1 renders zero hotels (up to 3 IPs; single attempt sans proxy)
- Persistent, pre-warmed context across dates so cache stays hot
  (~5 MB first page, ~0.3 MB per date after) instead of per-date cold loads
- Block images/media/fonts and third-party ad/consent/analytics hosts to
  cut bandwidth; never touch the anti-bot challenge (awswaf)
- Faster inter-page pacing now that a burned IP is cheap

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-05 20:30:15 +00:00
parent 1743590743
commit 11b995739c
2 changed files with 198 additions and 18 deletions

View file

@ -9,6 +9,12 @@ services:
- APP_SLUG=rates
- SETTINGS_URL=${SETTINGS_URL:-}
- SETTINGS_SECRET=${SETTINGS_SECRET:-}
# Residential proxy for the Booking.com scraper (empty = direct connection).
- BOOKING_PROXY_HOST=${BOOKING_PROXY_HOST:-}
- BOOKING_PROXY_PORT=${BOOKING_PROXY_PORT:-823}
- BOOKING_PROXY_USERNAME=${BOOKING_PROXY_USERNAME:-}
- BOOKING_PROXY_PASSWORD=${BOOKING_PROXY_PASSWORD:-}
- BOOKING_PROXY_COUNTRY=${BOOKING_PROXY_COUNTRY:-gb}
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 15s