Commit graph

1 commit

Author SHA1 Message Date
1dc8c0a945 Extract shared proxy module; make it available to direct scraper; remove dead code
Proxy config, DataImpulse sticky-session username syntax, and Playwright/
httpx proxy builders now live in one place (services/proxy.py) instead of
being duplicated across the Booking.com backend and the /config/proxy test
endpoint. Both scrapers consume it.

- services/proxy.py: load_config/normalize (DB-authoritative, env fallback),
  new_session_id, username, playwright_proxy, httpx_proxy_url
- PlaywrightLocalBackend delegates proxy building to the module
- get_scraper_backend factory uses proxy.load_config (one resolution path)
- test_proxy_config endpoint uses the shared URL builder; httpx proxies= ->
  proxy= (forward-compatible, 0.28-safe)
- Direct booking-engine scraper (httpx) can now route through the same proxy,
  gated by the direct_scraper_use_proxy flag (default off, plumbing ready)

Dead code removed: set_scraper_paused (never called — rotate-on-block
replaced pause-on-block), get_competitor_matrix / get_hotels_list /
update_hotel_tier (endpoints have their own SQL), unused PROXY_KEYS tuple.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 22:55:24 +00:00