hotel-manage-stack/rates
1
0
Fork 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Add Referer header to search URL navigation

Browse source 
A direct hit to searchresults.en-gb.html with no Referer causes Cloudflare
to hold the response body open (never sending HTML), so domcontentloaded
never fires. Setting referer=HOMEPAGE makes the request look like the user
searched from the homepage, which resolves the stall.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-09 17:50:45 +00:00
parent 3c4bc633fe
commit 7b4bb72f15
1 changed files with 6 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

7
backend/services/scraper_backends/playwright_local.py
View file

@ -592,7 +592,12 @@ class PlaywrightLocalBackend(ScraperBackend):
dest_id=dest_id, location_params=location_params)
logger.info(f"Scraping page {page_num + 1}: {url}")
try:
await page.goto(url, wait_until='domcontentloaded', timeout=90000)
# referer=HOMEPAGE makes the request look like the user
# searched from the homepage, which is required — a
# direct hit to searchresults with no Referer causes
# Cloudflare to hold the response body open indefinitely.
await page.goto(url, wait_until='domcontentloaded',
timeout=90000, referer=self.HOMEPAGE)
except Exception as e:
logger.warning(f"Page load timeout, continuing: {e}")
page_loaded = False