DataImpulse has two broken gateway nodes (.89, .97) and one working (.105).
Production DNS round-robin consistently landed on broken nodes. Fix is to
pin the proxy host to 67.213.121.105 in system_config. Revert to Chromium.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Booking.com detects headless Chromium via TLS ClientHello fingerprint (JA3)
and immediately flags the residential proxy IP on first contact. Firefox has
a different TLS fingerprint and GREASE extensions that aren't yet flagged.
Image blocking via Firefox preference (permissions.default.image=2) replaces
the Chrome --blink-settings flag. Dockerfile updated to install both browsers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
context.route() (request interception) breaks URL-embedded proxy auth in
Chromium — DataImpulse proactive CONNECT auth stops working when interception
is active. Replace route handler with --blink-settings=imagesEnabled=false
Chrome flag to block images without touching the interception layer.
Restore DataImpulse URL-embedded credentials (unchanged from before IPRoyal work).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
URL-embedded proxy credentials are dropped by Chromium when request
interception (context.route) is active. Both providers now use separate
username/password fields; DataImpulse incurs a 14s 407 round-trip on
context rotation but this only happens every 40 requests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse username contains semicolons (;sessid.ID) that weren't being
URL-encoded in httpx_proxy_url, causing silent parse failures. Also use
httpx.Proxy object instead of raw string (consistent with httpx_proxy()),
and capture repr(e) so empty-message exceptions show their type.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cf.bstatic.com (images + CSS CDN) was consuming 1.5 GB per scrape run.
Added route interception to abort image, stylesheet, font, and media
resources, plus pure tracking domains. The DOM extraction only reads
HTML attributes — no visual resources are needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
URL-embedded credentials are silently dropped by Chromium for IPRoyal
(proxy auth never sent → every page.goto times out). Separate username/
password fields work correctly — IPRoyal responds to the 407 challenge
quickly so there is no latency penalty unlike DataImpulse (~14s).
DataImpulse keeps URL-embedded credentials to avoid that round-trip.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auto-detect provider from hostname. DataImpulse appends session/country to
the username (LOGIN__cr.gb;sessid.ID); IPRoyal appends them to the password
(PASS_country-gb_session-ID_lifetime-30m). Both providers use the same
host/port/username/password/country config fields in Settings.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse endpoints are intermittently broken — some return EOF after
CONNECT immediately, others work fine. DNS round-robins between them so
the same request can fail 3× then succeed on the 4th. Timeouts were
returning blocked=False and silently failing with no retry. Now treated
as retryable blocks (with context rotation) so a fresh endpoint is tried.
Max retries 2→4 (5 total attempts) to give enough chances to land a
working DataImpulse endpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Header-based auth made no difference; the real issue is the hotel network
firewall blocking HTTPS CONNECT tunnels on port 823. Reverted to URL-embedded
credentials (original approach). Fix requires DataImpulse to enable port 443.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Embedding credentials in the proxy URL (http://user:pass@host:port) was
breaking HTTPS CONNECT tunnels on the hotel network. Switching to separate
username/password fields (Playwright) and httpx.Proxy(auth=...) sends a
Proxy-Authorization header instead, which passes through correctly.
With DataImpulse IP whitelisting the 407 round-trip is skipped anyway so
there is no latency penalty.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
asyncio.gather iterated undefined `shards` — should be `date_shards`
(variable renamed when switching from hotel-first to date-first sharding).
Caused every hotel-page scrape to immediately fail with NameError.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Workers now own a slice of dates and scrape all hotels per date before
advancing. A block or interruption leaves complete dates rather than
some hotels fully done and others not started.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old Stocks block_id has 0 in position 2 (not the persons count) so all
its rates were stored as max_persons=0 and deprioritised in the matrix.
Now reads max_persons from the visible 'Max persons: N' span (matching
the original scrapy spider), with aria-label/title fallback. block_id
segment 2 is unreliable across different hotel layouts.
Also treat max_persons=0 as unknown in the matrix ORDER BY priority.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add UA rotation (8 real Chrome UAs), random viewport, en-GB locale,
Europe/London timezone, stealth browser args to suppress automation
signals, and a per-context init script masking navigator.webdriver,
plugins, and chrome runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous approach walked siblings from [id^="room_type_id_"] anchors
and only processed rows with js-rt-block-row class, causing the first rate
row of some room types to be silently skipped when that class was absent.
Now iterates all #available_rooms tbody tr rows with data-block-id (same
strategy as the original scrapy spider), using the room_type_id_ element
presence to identify room names only on the first row, with fallback to the
stored name for subsequent rows of the same room type.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Include tier=own in get_active_hotels() so the own hotel Booking.com
listing is scraped alongside competitors. Change matrix max_persons
filter from hard WHERE to ORDER BY priority so hotels with only
1-person rates (e.g. Old Stocks) still show up rather than being
silently excluded.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_proxy_enabled() was checking cfg.get('server') but load_config() returns
{host, port, username, ...} — no 'server' key. Server URL is built by
proxy_util.playwright_proxy(). This meant proxy was silently disabled
and _proxy_kwargs() would also KeyError if called.
- _proxy_enabled() now delegates to proxy_util.is_enabled() (checks host+username)
- _proxy_kwargs() now calls proxy_util.playwright_proxy() with credentials
embedded in the URL (avoids 14s DataImpulse 407 round-trip)
- _get_context() generates a sticky session_id per context so each worker
gets a distinct residential IP lane
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The proxy-off → serial guard was correct for search-results (all workers
hit the same URL), but hotel-page workers each scrape a different hotel's
property page — parallel requests look like multi-tab browsing, not a
hammered aggregation endpoint. Added require_proxy param to
_effective_concurrency; hotel-page path passes require_proxy=False.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: restrict hotel-page scraper to 'competitor' tier only (was own+competitor); own hotel rates come from Newbook API, not Booking.com
- Backend: POST /competitors/discover — runs search-results scrape for one date to find market hotels regardless of current backend setting
- Backend: POST /competitors/hotels — add hotel manually from Booking.com URL + name + tier; upserts on slug conflict
- Frontend (Settings tab): Discover Market Hotels button added below manual date-range scrape
- Frontend (Hotels tab): Add Hotel form at top — paste URL (auto-derives name from slug), choose tier, submit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Market-tier hotels were auto-discovered from search results and don't
need room-level rate tracking — scraping all 25+ of them per date was
unnecessary. Only 'own' and 'competitor' hotels are now scraped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: extract max_persons from block_id (3rd segment) instead of cell text; cell[0] was picking up the price value (e.g. 189/166) causing the max_persons=2 filter to exclude all scraped rows and show "No rates available" in snapshot
- MarketView: add namelength: -1 to Plotly legend to prevent label truncation; increase bottom margin from 50→65 and legend y from -0.25→-0.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'We have X left' scarcity indicator only appears when availability is
low (typically ≤5). The quantity <select> dropdown in the booking form
always shows 0..N where N = actual available qty (capped at 10).
Extract max option value from the <select> in the last table cell per rate
plan row, falling back to regex parsing of the cell text if the select
isn't rendered, then to the scarcity text as a last resort.
Stored in both rooms_left and available_qty on RateData.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switches booking.com scraping from search-results page (Cloudflare-targeted)
to individual hotel property pages (not CF-protected). Each page load returns
all room types, all rate plan variants (room-only/B&B × refundable/non-ref ×
1-2 adults), and availability counts.
Key changes:
- New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF
- booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(),
_scrape_hotels_concurrent() — sharded by hotel so one proxy session covers
all dates for one hotel (looks human)
- schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates
- competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker
so DISTINCT ON returns cheapest 2-adult rate from latest batch
Enable via Settings → Scraper Backend → playwright_hotel_page
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When page.goto() times out while Cloudflare's JS challenge redirect is in
progress, page.content() silently blocks until that navigation completes
before raising — adding ~1m46s of hidden delay per blocked-page attempt.
Fix: skip page.content() entirely when page_loaded=False (goto already
timed out), setting content="" so block detection treats it as no signal.
Also break out of the page loop early when page 1 failed to load with 0
hotels — no point spending another 60s on the offset-URL page 2 when the
browser is in a Cloudflare challenge state.
Per-blocked-attempt time: ~4 min → ~1 min (30s goto + 30s selectors).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Timeouts were set for the old proxy 407 round-trip world (14.5s latency
per challenge). Now that credentials are embedded in the proxy URL, pages
load in ~7s — the 90s goto timeout was forcing 5+ minutes of wasted wait
on Cloudflare-blocked dates before rotating sessions.
homepage warmup goto: 90s → 20s
page 1 search goto: 90s → 30s (4× normal load time headroom)
property-card waits: 30s → 15s (cards appear in <3s on clean pages)
Default concurrency 3 → 6: each worker uses its own residential proxy IP
so parallelism is safe. ~0.4 GB per Chromium; 6 workers fits in 4 GB LXC.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse takes ~14s to issue a 407 challenge from this network.
Passing credentials as separate Playwright proxy fields caused Chromium
to wait for that challenge before sending auth on every CONNECT request,
making scrapes consistently time out at 90s.
Embedding credentials directly in the proxy server URL makes Chromium
send Proxy-Authorization on the first CONNECT, bypassing the round-trip.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
page.content() throws when page is still navigating after a goto timeout;
catch it and continue with empty content. Also broaden the soft-block check
to trigger IP rotation whenever no hotels are found (not just on explicit
blocks), so proxy rotation fires even when the error path is hit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When all 3 proxy rotation attempts are soft-blocked (bad IP pool), fall
back to the server's own IP for a final attempt. Keeps proxy as primary
for IP diversity / rate-limit protection while ensuring a clean fallback
when the assigned residential pool is Cloudflare-challenged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A direct hit to searchresults.en-gb.html with no Referer causes Cloudflare
to hold the response body open (never sending HTML), so domcontentloaded
never fires. Setting referer=HOMEPAGE makes the request look like the user
searched from the homepage, which resolves the stall.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Headless mode gets blocked by Cloudflare on the search endpoint. Headed
mode with Xvfb works on dev (Unraid). The missing piece on Proxmox LXC was
insufficient /dev/shm (Docker default 64 MB); setting shm_size: 256m gives
Xvfb enough shared memory to start.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse residential proxy takes ~29s to route through Booking.com's
Cloudflare edge — the previous 30s timeout was too short and the page
never had a chance to deliver its body. Also raised wait_for_selector
from 15s to 30s so property cards have time to render post-load.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Xvfb exits immediately on the production LXC (no framebuffer device
support), so headed mode was never working. Switch to headless=True
and remove Xvfb from the Dockerfile entirely.
Stealth coverage is unchanged: playwright-stealth patches webdriver,
plugins and chrome.runtime; the residential proxy + sticky session
provide the GB residential fingerprint; homepage warmup carries real
session cookies into the search. --enable-unsafe-swiftshader and
--disable-blink-features=AutomationControlled are kept.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: threading.Lock held indefinitely when Playwright browser
hangs inside run_in_executor (finally never fires from the async side).
Fixes:
- _acquire_scrape_lock/_release_scrape_lock track monotonic timestamp
- POST /competitors/scrape/reset force-releases the lock and marks any
running batch as interrupted (queue rows stay intact for retry)
- GET /competitors/status now includes lock_held_seconds
- APScheduler watchdog job every 30 min auto-releases if held >3h
- Settings → Scraper Proxy tab shows live lock status (green/amber)
with a Force Reset button requiring confirmation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Direct-rates sub-row in Market View was dead: the frontend filters
competitors on direct_hotel_id but /matrix and /hotels never returned it.
Add direct_hotel_id to both queries (+ HotelResponse), and only render the
"Direct" sub-row when a hotel actually has a non-null direct rate (was
rendering all-dashes on an empty {} object).
Remove the pause-on-block flow entirely — dormant since rotate-on-block
replaced it (nothing set booking_scraper_paused=true after set_scraper_paused
was dropped): is_scraper_paused, /config/unpause, the /scrape paused guard,
ScraperStatusResponse.paused/pause_until, and the frontend Paused badge +
Unpause button. Trim now-unused datetime import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Proxy config, DataImpulse sticky-session username syntax, and Playwright/
httpx proxy builders now live in one place (services/proxy.py) instead of
being duplicated across the Booking.com backend and the /config/proxy test
endpoint. Both scrapers consume it.
- services/proxy.py: load_config/normalize (DB-authoritative, env fallback),
new_session_id, username, playwright_proxy, httpx_proxy_url
- PlaywrightLocalBackend delegates proxy building to the module
- get_scraper_backend factory uses proxy.load_config (one resolution path)
- test_proxy_config endpoint uses the shared URL builder; httpx proxies= ->
proxy= (forward-compatible, 0.28-safe)
- Direct booking-engine scraper (httpx) can now route through the same proxy,
gated by the direct_scraper_use_proxy flag (default off, plumbing ready)
Dead code removed: set_scraper_paused (never called — rotate-on-block
replaced pause-on-block), get_competitor_matrix / get_hotels_list /
update_hotel_tier (endpoints have their own SQL), unused PROXY_KEYS tuple.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fan a scrape run out across N workers, each with its own DB session and
its own scraper backend — and since every backend picks a random sticky
session id, each worker scrapes from a distinct residential IP. Dates are
interleaved across workers so each covers a spread of the range. Cuts a
150-date batch from ~30-45 min to ~12-15 min at 3 workers.
- booking_scraper_concurrency config key (default 3); Settings → Scraper
Proxy has a "Parallel workers" field
- Forced to 1 when the proxy is off (N workers would share one IP and
hammer it) or when there's a single date
- Both manual and queue paths routed through _scrape_dates_concurrent;
workers mark their own queue items
- Per-worker rotate-on-block replaces the old global pause-on-block
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The location_search_url column existed but was dead — the scraper always
rebuilt the URL from the free-text location name. Now the Location
Configuration form takes a "Booking.com search URL" field: paste the
address-bar URL from a real search and the scraper lifts ss/dest_id/
dest_type from it (the most reliable destination pin). Falls back to
dest_id, then plain name. Server also extracts dest_id from the URL for
the column and derives a display name from ss when none is typed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Free-text ss= destination resolution is non-deterministic: tonight's
30-day run resolved "Stow on the Wold" to St. Wolfgang, Austria for 8 of
30 dates, saving Salzkammergut hotel rates into the matrix. dest_id +
dest_type=city in the search URL pins the destination.
- booking_scrape_config gains a dest_id column (idempotent ALTER)
- scrape_location_search/_build_search_url thread dest_id through
- /config/location accepts dest_id
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "Scraper Proxy" tab: enable toggle, host/port/username/password/
country, Save, and a Test Connection button that reports the live exit
IP + country through the proxy
- Backend proxy config now lives in system_config (DB authoritative when
booking_proxy_enabled is set; BOOKING_PROXY_* env vars are the fallback)
- Dedicated /config/proxy GET/POST/test endpoints; password is write-only
(never returned, blank keeps the stored value) and masked in /config/system
- Surface proxy status keys in the read-only System tab
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Route through DataImpulse residential proxy when BOOKING_PROXY_* env
vars are set (empty = direct connection, unchanged behaviour)
- Sticky one IP per session via DataImpulse sessid; rotate_session()
burns the IP for a fresh one
- Rotate-on-block: retry a date on a new IP when the page is challenged
or page 1 renders zero hotels (up to 3 IPs; single attempt sans proxy)
- Persistent, pre-warmed context across dates so cache stays hot
(~5 MB first page, ~0.3 MB per date after) instead of per-date cold loads
- Block images/media/fonts and third-party ad/consent/analytics hosts to
cut bandwidth; never touch the anti-bot challenge (awswaf)
- Faster inter-page pacing now that a burned IP is cheap
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page-load timeout was logged and skipped, so a scrape could 'succeed'
with only page 1 of results — and the not_listed flagging then marked
every page-2 hotel absent, suppressing their last known rates.
- ScraperResult now tracks pages_requested/pages_ok
- scrape_date skips not_listed flagging when pages failed or when the
scrape saw <60% of the date's 7-day coverage baseline; scraped rates
are still saved, unseen hotels keep last known rate + scrape time
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Location-search results aren't a fixed hotel set — a sold-out hotel drops
out and its last 'available' rate would remain the latest row for that
date, reading as a live price and skewing market averages. On each
successful per-date scrape, insert a NULL-rate 'not_listed' row for every
active hotel missing from the results (skipped if the parse found nothing,
which indicates scraper fault not absence). Failed/blocked scrapes write
nothing, so genuinely-stale data remains distinguishable by scraped_at.
Also: fix the DOW analysis to pick latest-then-filter so a not_listed
latest row drops the date instead of resurfacing an older rate, and widen
booking_com_id to VARCHAR(255) (some Booking slugs exceed 50 chars).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scraped room descriptions exceed VARCHAR(100) (StringDataRightTruncation),
and the failed insert poisoned the transaction so every subsequent save in
the batch died with InFailedSqlTransaction — scrapes reported success with
0 rows saved. Widen the column (with ALTER for existing tables) and roll
back after a failed save.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bookability showed no availability because newbook_occupancy_report_data
was never populated: add reports_occupancy client method and
sync_occupancy job, run before rates in both Sync Now and the daily
schedule (single fast API call)
- Category order: default display_order to the Newbook category id on
sync (was 0 → alphabetical), preserve manual order on re-sync, extend
PATCH to accept display_order, add up/down reorder arrows in Settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rates Newbook instance returns categories as category_id/category_name
on each site; also skip inactive categories.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- require_cap was a Depends-factory but every call site uses it inline;
make it an inline checker (fixes 500 on /analysis/hotels, /direct/*)
- /analysis/comparison returned a per-date matrix the frontend never read;
return per-hotel aggregates (our/their avg, price index) and default to
all active competitors so the Market Comparison table works without params
- Room categories were never populated (lost in port): add sites_list fetch
to the Newbook client, categories list/sync/toggle endpoints, and a
Settings card — without included categories every rates sync exits early
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
