page.content() throws when page is still navigating after a goto timeout; catch it and continue with empty content. Also broaden the soft-block check to trigger IP rotation whenever no hotels are found (not just on explicit blocks), so proxy rotation fires even when the error path is hit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| direct_profiles | ||
| scraper_backends | ||
| __init__.py | ||
| booking_scraper.py | ||
| central_settings.py | ||
| direct_scraper.py | ||
| newbook_rates_client.py | ||
| proxy.py | ||