jtricerolph
6b7f00b40a
Fan a scrape run out across N workers, each with its own DB session and its own scraper backend — and since every backend picks a random sticky session id, each worker scrapes from a distinct residential IP. Dates are interleaved across workers so each covers a spread of the range. Cuts a 150-date batch from ~30-45 min to ~12-15 min at 3 workers. - booking_scraper_concurrency config key (default 3); Settings → Scraper Proxy has a "Parallel workers" field - Forced to 1 when the proxy is off (N workers would share one IP and hammer it) or when there's a single date - Both manual and queue paths routed through _scrape_dates_concurrent; workers mark their own queue items - Per-worker rotate-on-block replaces the old global pause-on-block Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| direct_profiles | ||
| scraper_backends | ||
| __init__.py | ||
| booking_scraper.py | ||
| central_settings.py | ||
| direct_scraper.py | ||
| newbook_rates_client.py | ||