Add hotel-page scraper backend with full room/rate plan extraction

Switches booking.com scraping from search-results page (Cloudflare-targeted)
to individual hotel property pages (not CF-protected). Each page load returns
all room types, all rate plan variants (room-only/B&B × refundable/non-ref ×
1-2 adults), and availability counts.

Key changes:
- New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF
- booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(),
  _scrape_hotels_concurrent() — sharded by hotel so one proxy session covers
  all dates for one hotel (looks human)
- schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates
- competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker
  so DISTINCT ON returns cheapest 2-adult rate from latest batch

Enable via Settings → Scraper Backend → playwright_hotel_page

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-09 23:08:03 +00:00
parent b10b2f5990
commit bb37fcf501
6 changed files with 608 additions and 8 deletions

View file

@ -675,7 +675,8 @@ async def get_competitor_matrix(
WHERE {tier_filter}
AND h.is_active = TRUE
AND r.rate_date >= :from_date AND r.rate_date <= :to_date
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC
AND (r.max_persons IS NULL OR r.max_persons = 2)
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC, r.rate_gross ASC NULLS LAST
"""),
{'from_date': start, 'to_date': end}
)
@ -714,7 +715,8 @@ async def get_competitor_matrix(
AND r.rate_date >= :from_date AND r.rate_date <= :to_date
AND r.availability_status = 'available'
AND r.rate_gross IS NOT NULL
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC
AND (r.max_persons IS NULL OR r.max_persons = 2)
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC, r.rate_gross ASC NULLS LAST
"""),
{'from_date': start, 'to_date': end}
)