hotel-manage-stack/rates
1
0
Fork 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Fix page.content() blocking ~2 minutes after goto timeout on Cloudflare hits

Browse source 
When page.goto() times out while Cloudflare's JS challenge redirect is in
progress, page.content() silently blocks until that navigation completes
before raising — adding ~1m46s of hidden delay per blocked-page attempt.

Fix: skip page.content() entirely when page_loaded=False (goto already
timed out), setting content="" so block detection treats it as no signal.

Also break out of the page loop early when page 1 failed to load with 0
hotels — no point spending another 60s on the offset-URL page 2 when the
browser is in a Cloudflare challenge state.

Per-blocked-attempt time: ~4 min → ~1 min (30s goto + 30s selectors).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-09 21:34:37 +00:00
parent ea66025140
commit b10b2f5990
1 changed files with 15 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

22
backend/services/scraper_backends/playwright_local.py
View file

@ -636,13 +636,15 @@ class PlaywrightLocalBackend(ScraperBackend):
logger.warning(f"Page load timeout, continuing: {e}") logger.warning(f"Page load timeout, continuing: {e}")
page_loaded = False page_loaded = False
# Check for blocking — page may still be navigating after a # Check for blocking. Skip when the page never loaded — page.content()
# timeout, so wrap in try/except to avoid crashing the attempt. # blocks until any in-progress navigation (e.g. Cloudflare redirect)
try: # completes before raising, adding minutes of hidden delay per attempt.
content = await page.content() content = ""
except Exception as ce: if page_loaded:
logger.warning(f"page.content() unavailable (page still navigating): {ce}") try:
content = "" content = await page.content()
except Exception as ce:
logger.warning(f"page.content() unavailable: {ce}")
is_blocked, reason = self.detect_blocking(content) is_blocked, reason = self.detect_blocking(content)
if is_blocked: if is_blocked:
logger.warning(f"Blocking detected: {reason}") logger.warning(f"Blocking detected: {reason}")
@ -679,6 +681,12 @@ class PlaywrightLocalBackend(ScraperBackend):
logger.info(f"Page {page_num + 1}: found {len(hotels)} hotels, {len(all_hotels)} total unique") logger.info(f"Page {page_num + 1}: found {len(hotels)} hotels, {len(all_hotels)} total unique")
# Page 1 failed to load and returned nothing — the browser is in
# a broken/challenge state; trying page 2 just wastes another 60s.
if page_num == 0 and not page_loaded and not all_hotels:
logger.warning("Page 1 load failed with 0 hotels — skipping page 2")
break
return ScraperResult( return ScraperResult(
success=True, success=True,
blocked=False, blocked=False,