rates/backend/services
jtricerolph ea66025140 Reduce scraper timeouts and raise default concurrency to 6
Timeouts were set for the old proxy 407 round-trip world (14.5s latency
per challenge). Now that credentials are embedded in the proxy URL, pages
load in ~7s — the 90s goto timeout was forcing 5+ minutes of wasted wait
on Cloudflare-blocked dates before rotating sessions.

  homepage warmup goto: 90s → 20s
  page 1 search goto:   90s → 30s  (4× normal load time headroom)
  property-card waits:  30s → 15s  (cards appear in <3s on clean pages)

Default concurrency 3 → 6: each worker uses its own residential proxy IP
so parallelism is safe. ~0.4 GB per Chromium; 6 workers fits in 4 GB LXC.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-09 21:17:19 +00:00
..
direct_profiles Add Rate Monitor app — Booking.com + direct booking engine competitor rates 2026-07-05 12:06:30 +00:00
scraper_backends Reduce scraper timeouts and raise default concurrency to 6 2026-07-09 21:17:19 +00:00
__init__.py Add Rate Monitor app — Booking.com + direct booking engine competitor rates 2026-07-05 12:06:30 +00:00
booking_scraper.py Reduce scraper timeouts and raise default concurrency to 6 2026-07-09 21:17:19 +00:00
central_settings.py Add Rate Monitor app — Booking.com + direct booking engine competitor rates 2026-07-05 12:06:30 +00:00
direct_scraper.py Extract shared proxy module; make it available to direct scraper; remove dead code 2026-07-05 22:55:24 +00:00
newbook_rates_client.py Port occupancy sync + category ordering from the forecasting version 2026-07-05 14:18:03 +00:00
proxy.py Fix proxy 407 round-trip causing 14s latency per page.goto() 2026-07-09 20:14:09 +00:00