jtricerolph
b10b2f5990
When page.goto() times out while Cloudflare's JS challenge redirect is in progress, page.content() silently blocks until that navigation completes before raising — adding ~1m46s of hidden delay per blocked-page attempt. Fix: skip page.content() entirely when page_loaded=False (goto already timed out), setting content="" so block detection treats it as no signal. Also break out of the page loop early when page 1 failed to load with 0 hotels — no point spending another 60s on the offset-URL page 2 when the browser is in a Cloudflare challenge state. Per-blocked-attempt time: ~4 min → ~1 min (30s goto + 30s selectors). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| api | ||
| jobs | ||
| services | ||
| auth.py | ||
| database.py | ||
| Dockerfile | ||
| main.py | ||
| migrate_direct_data.py | ||
| requirements.txt | ||
| scheduler.py | ||
| schema.sql | ||