hotel-manage-stack/rates
1
0
Fork 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Scraper: process-wide lock, one scrape at a time

Browse source 
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-05 16:14:55 +00:00
parent bf9425ee7e
commit b712196262
3 changed files with 56 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

6
backend/api/competitors.py
View file

@ -297,6 +297,12 @@ async def trigger_manual_scrape(
if paused_row and paused_row.config_value == 'true':
raise HTTPException(status_code=400, detail="Scraper is currently paused. Use /unpause first or wait for cooldown.")
# Only one scrape at a time — concurrent Chromium runs cause the page
# timeouts that produce partial results
from services.booking_scraper import SCRAPE_LOCK
if SCRAPE_LOCK.locked():
raise HTTPException(status_code=409, detail="A scrape is already running. Try again when it finishes.")
# Start background task
background_tasks.add_task(run_scrape_sync, from_date, to_date)