jtricerolph
6b7f00b40a
Fan a scrape run out across N workers, each with its own DB session and its own scraper backend — and since every backend picks a random sticky session id, each worker scrapes from a distinct residential IP. Dates are interleaved across workers so each covers a spread of the range. Cuts a 150-date batch from ~30-45 min to ~12-15 min at 3 workers. - booking_scraper_concurrency config key (default 3); Settings → Scraper Proxy has a "Parallel workers" field - Forced to 1 when the proxy is off (N workers would share one IP and hammer it) or when there's a single date - Both manual and queue paths routed through _scrape_dates_concurrent; workers mark their own queue items - Per-worker rotate-on-block replaces the old global pause-on-block Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| api | ||
| jobs | ||
| services | ||
| auth.py | ||
| database.py | ||
| Dockerfile | ||
| main.py | ||
| migrate_direct_data.py | ||
| requirements.txt | ||
| scheduler.py | ||
| schema.sql | ||