hotel-manage-stack/rates
1
0
Fork 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Retry page load timeouts: treat as blocks, bump max retries to 4

Browse source 
DataImpulse endpoints are intermittently broken — some return EOF after
CONNECT immediately, others work fine. DNS round-robins between them so
the same request can fail 3× then succeed on the 4th. Timeouts were
returning blocked=False and silently failing with no retry. Now treated
as retryable blocks (with context rotation) so a fresh endpoint is tried.
Max retries 2→4 (5 total attempts) to give enough chances to land a
working DataImpulse endpoint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-10 02:19:16 +00:00
parent 28c4dd9f54
commit b44e688b3e
2 changed files with 6 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

2
backend/services/booking_scraper.py
View file

@ -318,7 +318,7 @@ async def scrape_hotel_date(
backend: 'PlaywrightHotelPageBackend', backend: 'PlaywrightHotelPageBackend',
batch_id: uuid.UUID, batch_id: uuid.UUID,
adults: int = 2, adults: int = 2,
max_retries: int = 2, max_retries: int = 4,
) -> Dict[str, Any]: ) -> Dict[str, Any]:
""" """
Scrape all rate plans for one hotel on one date via the hotel page backend. Scrape all rate plans for one hotel on one date via the hotel page backend.

6
backend/services/scraper_backends/playwright_hotel_page.py
View file

@ -298,7 +298,11 @@ class PlaywrightHotelPageBackend(ScraperBackend):
logger.warning(f"goto timeout for {hotel_url} {check_in}: {e}") logger.warning(f"goto timeout for {hotel_url} {check_in}: {e}")
if not page_loaded: if not page_loaded:
return ScraperResult(success=False, blocked=False, error_message='page load timeout') # Treat as a retryable block — DataImpulse endpoints are intermittently
# broken (EOF after CONNECT); rotating to a fresh context hits a different
# endpoint and usually succeeds within 1-2 retries.
await self._rotate_context()
return ScraperResult(success=False, blocked=True, block_reason='page load timeout')
# Wait for the room table (prices render via JS after DOM) # Wait for the room table (prices render via JS after DOM)
room_table_appeared = False room_table_appeared = False