Add hotel-page scraper backend with full room/rate plan extraction
Switches booking.com scraping from search-results page (Cloudflare-targeted) to individual hotel property pages (not CF-protected). Each page load returns all room types, all rate plan variants (room-only/B&B × refundable/non-ref × 1-2 adults), and availability counts. Key changes: - New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF - booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(), _scrape_hotels_concurrent() — sharded by hotel so one proxy session covers all dates for one hotel (looks human) - schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates - competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker so DISTINCT ON returns cheapest 2-adult rate from latest batch Enable via Settings → Scraper Backend → playwright_hotel_page Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
b10b2f5990
commit
bb37fcf501
6 changed files with 608 additions and 8 deletions
|
|
@ -9,6 +9,7 @@ Provides pluggable backends to allow switching between:
|
|||
|
||||
from .base import ScraperBackend, ScraperResult, HotelData, RateData, AvailabilityStatus
|
||||
from .playwright_local import PlaywrightLocalBackend
|
||||
from .playwright_hotel_page import PlaywrightHotelPageBackend
|
||||
|
||||
__all__ = [
|
||||
'ScraperBackend',
|
||||
|
|
@ -17,4 +18,5 @@ __all__ = [
|
|||
'RateData',
|
||||
'AvailabilityStatus',
|
||||
'PlaywrightLocalBackend',
|
||||
'PlaywrightHotelPageBackend',
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue