cf.bstatic.com (images + CSS CDN) was consuming 1.5 GB per scrape run.
Added route interception to abort image, stylesheet, font, and media
resources, plus pure tracking domains. The DOM extraction only reads
HTML attributes — no visual resources are needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse endpoints are intermittently broken — some return EOF after
CONNECT immediately, others work fine. DNS round-robins between them so
the same request can fail 3× then succeed on the 4th. Timeouts were
returning blocked=False and silently failing with no retry. Now treated
as retryable blocks (with context rotation) so a fresh endpoint is tried.
Max retries 2→4 (5 total attempts) to give enough chances to land a
working DataImpulse endpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old Stocks block_id has 0 in position 2 (not the persons count) so all
its rates were stored as max_persons=0 and deprioritised in the matrix.
Now reads max_persons from the visible 'Max persons: N' span (matching
the original scrapy spider), with aria-label/title fallback. block_id
segment 2 is unreliable across different hotel layouts.
Also treat max_persons=0 as unknown in the matrix ORDER BY priority.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add UA rotation (8 real Chrome UAs), random viewport, en-GB locale,
Europe/London timezone, stealth browser args to suppress automation
signals, and a per-context init script masking navigator.webdriver,
plugins, and chrome runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous approach walked siblings from [id^="room_type_id_"] anchors
and only processed rows with js-rt-block-row class, causing the first rate
row of some room types to be silently skipped when that class was absent.
Now iterates all #available_rooms tbody tr rows with data-block-id (same
strategy as the original scrapy spider), using the room_type_id_ element
presence to identify room names only on the first row, with fallback to the
stored name for subsequent rows of the same room type.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_proxy_enabled() was checking cfg.get('server') but load_config() returns
{host, port, username, ...} — no 'server' key. Server URL is built by
proxy_util.playwright_proxy(). This meant proxy was silently disabled
and _proxy_kwargs() would also KeyError if called.
- _proxy_enabled() now delegates to proxy_util.is_enabled() (checks host+username)
- _proxy_kwargs() now calls proxy_util.playwright_proxy() with credentials
embedded in the URL (avoids 14s DataImpulse 407 round-trip)
- _get_context() generates a sticky session_id per context so each worker
gets a distinct residential IP lane
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: extract max_persons from block_id (3rd segment) instead of cell text; cell[0] was picking up the price value (e.g. 189/166) causing the max_persons=2 filter to exclude all scraped rows and show "No rates available" in snapshot
- MarketView: add namelength: -1 to Plotly legend to prevent label truncation; increase bottom margin from 50→65 and legend y from -0.25→-0.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'We have X left' scarcity indicator only appears when availability is
low (typically ≤5). The quantity <select> dropdown in the booking form
always shows 0..N where N = actual available qty (capped at 10).
Extract max option value from the <select> in the last table cell per rate
plan row, falling back to regex parsing of the cell text if the select
isn't rendered, then to the scarcity text as a last resort.
Stored in both rooms_left and available_qty on RateData.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switches booking.com scraping from search-results page (Cloudflare-targeted)
to individual hotel property pages (not CF-protected). Each page load returns
all room types, all rate plan variants (room-only/B&B × refundable/non-ref ×
1-2 adults), and availability counts.
Key changes:
- New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF
- booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(),
_scrape_hotels_concurrent() — sharded by hotel so one proxy session covers
all dates for one hotel (looks human)
- schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates
- competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker
so DISTINCT ON returns cheapest 2-adult rate from latest batch
Enable via Settings → Scraper Backend → playwright_hotel_page
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
