rates/backend/services/scraper_backends
jtricerolph ea66025140 Reduce scraper timeouts and raise default concurrency to 6
Timeouts were set for the old proxy 407 round-trip world (14.5s latency
per challenge). Now that credentials are embedded in the proxy URL, pages
load in ~7s — the 90s goto timeout was forcing 5+ minutes of wasted wait
on Cloudflare-blocked dates before rotating sessions.

  homepage warmup goto: 90s → 20s
  page 1 search goto:   90s → 30s  (4× normal load time headroom)
  property-card waits:  30s → 15s  (cards appear in <3s on clean pages)

Default concurrency 3 → 6: each worker uses its own residential proxy IP
so parallelism is safe. ~0.4 GB per Chromium; 6 workers fits in 4 GB LXC.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-09 21:17:19 +00:00
..
__init__.py Add Rate Monitor app — Booking.com + direct booking engine competitor rates 2026-07-05 12:06:30 +00:00
base.py Use pasted Booking.com search URL to pin scrape destination 2026-07-05 22:00:28 +00:00
playwright_local.py Reduce scraper timeouts and raise default concurrency to 6 2026-07-09 21:17:19 +00:00