rates/backend
jtricerolph ea66025140 Reduce scraper timeouts and raise default concurrency to 6
Timeouts were set for the old proxy 407 round-trip world (14.5s latency
per challenge). Now that credentials are embedded in the proxy URL, pages
load in ~7s — the 90s goto timeout was forcing 5+ minutes of wasted wait
on Cloudflare-blocked dates before rotating sessions.

  homepage warmup goto: 90s → 20s
  page 1 search goto:   90s → 30s  (4× normal load time headroom)
  property-card waits:  30s → 15s  (cards appear in <3s on clean pages)

Default concurrency 3 → 6: each worker uses its own residential proxy IP
so parallelism is safe. ~0.4 GB per Chromium; 6 workers fits in 4 GB LXC.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-09 21:17:19 +00:00
..
api Fix scrape submission race condition allowing duplicate background tasks 2026-07-09 20:59:17 +00:00
jobs Parity: best-available comparison as default, term-matching optional 2026-07-06 08:04:15 +00:00
services Reduce scraper timeouts and raise default concurrency to 6 2026-07-09 21:17:19 +00:00
auth.py Fix require_cap TypeError, rewrite /analysis/comparison, add room category sync 2026-07-05 13:47:10 +00:00
database.py
Dockerfile Revert to headed Chrome + Xvfb; add shm_size 256m to fix Xvfb in Docker 2026-07-09 17:36:27 +00:00
main.py
migrate_direct_data.py Restore guestline-monitor features: room stock/occupancy, rate history, room stats 2026-07-06 06:50:37 +00:00
requirements.txt Scraper stealth: headful+xvfb, playwright-stealth, coherent UA, warm-up, click-paginate 2026-07-05 16:26:01 +00:00
scheduler.py Scraper: force-reset endpoint + watchdog to prevent stuck lock 2026-07-09 16:56:35 +00:00
schema.sql Pin Booking.com search to a dest_id 2026-07-05 21:49:57 +00:00