hotel-manage-stack/rates
1
0
Fork 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Add hotel-page scraper backend with full room/rate plan extraction

Browse source 
Switches booking.com scraping from search-results page (Cloudflare-targeted)
to individual hotel property pages (not CF-protected). Each page load returns
all room types, all rate plan variants (room-only/B&B × refundable/non-ref ×
1-2 adults), and availability counts.

Key changes:
- New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF
- booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(),
  _scrape_hotels_concurrent() — sharded by hotel so one proxy session covers
  all dates for one hotel (looks human)
- schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates
- competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker
  so DISTINCT ON returns cheapest 2-adult rate from latest batch

Enable via Settings → Scraper Backend → playwright_hotel_page

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-09 23:08:03 +00:00
parent b10b2f5990
commit bb37fcf501
6 changed files with 608 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

6
backend/schema.sql
View file

@ -17,7 +17,7 @@ INSERT INTO system_config (config_key, config_value, description) VALUES
('booking_scraper_enabled', 'false', 'Enable automatic booking.com rate scraping (true/false)'),
('booking_scraper_paused', 'false', 'Scraper temporarily paused due to blocking (true/false)'),
('booking_scraper_pause_until', NULL, 'ISO datetime when pause expires'),
('booking_scraper_backend', 'playwright_local', 'Scraper backend: playwright_local | playwright_proxy | apify'),
('booking_scraper_backend', 'playwright_local', 'Scraper backend: playwright_local | playwright_hotel_page | apify'),
('booking_scraper_daily_time', '05:30', 'Daily scrape time (HH:MM)'),
('booking_scraper_proxy_url', NULL, 'Proxy URL for playwright_proxy backend'),
('booking_scraper_proxy_username', NULL, 'Proxy username for playwright_proxy backend'),
@ -128,6 +128,10 @@ CREATE TABLE IF NOT EXISTS booking_com_rates (
ALTER TABLE booking_com_rates ALTER COLUMN room_type TYPE TEXT;
ALTER TABLE booking_com_hotels ALTER COLUMN booking_com_id TYPE VARCHAR(255);
-- Hotel page scraper: rate plan identifier and occupancy per row
ALTER TABLE booking_com_rates ADD COLUMN IF NOT EXISTS rate_plan_id TEXT;
ALTER TABLE booking_com_rates ADD COLUMN IF NOT EXISTS max_persons INTEGER;
CREATE INDEX IF NOT EXISTS idx_booking_com_rates_hotel_date ON booking_com_rates(hotel_id, rate_date);
CREATE INDEX IF NOT EXISTS idx_booking_com_rates_date ON booking_com_rates(rate_date);
CREATE INDEX IF NOT EXISTS idx_booking_com_rates_scraped ON booking_com_rates(scraped_at DESC);