Add hotel-page scraper backend with full room/rate plan extraction
Switches booking.com scraping from search-results page (Cloudflare-targeted) to individual hotel property pages (not CF-protected). Each page load returns all room types, all rate plan variants (room-only/B&B × refundable/non-ref × 1-2 adults), and availability counts. Key changes: - New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF - booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(), _scrape_hotels_concurrent() — sharded by hotel so one proxy session covers all dates for one hotel (looks human) - schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates - competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker so DISTINCT ON returns cheapest 2-adult rate from latest batch Enable via Settings → Scraper Backend → playwright_hotel_page Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
b10b2f5990
commit
bb37fcf501
6 changed files with 608 additions and 8 deletions
|
|
@ -675,7 +675,8 @@ async def get_competitor_matrix(
|
|||
WHERE {tier_filter}
|
||||
AND h.is_active = TRUE
|
||||
AND r.rate_date >= :from_date AND r.rate_date <= :to_date
|
||||
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC
|
||||
AND (r.max_persons IS NULL OR r.max_persons = 2)
|
||||
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC, r.rate_gross ASC NULLS LAST
|
||||
"""),
|
||||
{'from_date': start, 'to_date': end}
|
||||
)
|
||||
|
|
@ -714,7 +715,8 @@ async def get_competitor_matrix(
|
|||
AND r.rate_date >= :from_date AND r.rate_date <= :to_date
|
||||
AND r.availability_status = 'available'
|
||||
AND r.rate_gross IS NOT NULL
|
||||
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC
|
||||
AND (r.max_persons IS NULL OR r.max_persons = 2)
|
||||
ORDER BY r.hotel_id, r.rate_date, r.scraped_at DESC, r.rate_gross ASC NULLS LAST
|
||||
"""),
|
||||
{'from_date': start, 'to_date': end}
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue