- Route through DataImpulse residential proxy when BOOKING_PROXY_* env
vars are set (empty = direct connection, unchanged behaviour)
- Sticky one IP per session via DataImpulse sessid; rotate_session()
burns the IP for a fresh one
- Rotate-on-block: retry a date on a new IP when the page is challenged
or page 1 renders zero hotels (up to 3 IPs; single attempt sans proxy)
- Persistent, pre-warmed context across dates so cache stays hot
(~5 MB first page, ~0.3 MB per date after) instead of per-date cold loads
- Block images/media/fonts and third-party ad/consent/analytics hosts to
cut bandwidth; never touch the anti-bot challenge (awswaf)
- Faster inter-page pacing now that a burned IP is cheap
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
xvfb-run as container PID 1 hangs before exec'ing uvicorn — its Xvfb
readiness handshake (SIGUSR1) is never delivered to PID 1, so the backend
sat 'Up (unhealthy)' with no logs and the frontend never started.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tier 1+2 anti-detection to stop Booking.com throttling page 2:
- Run Chromium headful under xvfb (Dockerfile) — headless leaks SwiftShader
WebGL, empty plugins, missing chrome.runtime
- playwright-stealth patches navigator.webdriver/plugins/WebGL vendor
- Single coherent Chrome-121 identity: UA + matching sec-ch-ua client hints +
platform (dropped the Firefox/Safari UA strings — a mismatched UA on a
Chromium engine is a stronger tell than no rotation)
- Homepage warm-up so search requests carry real session cookies + cookie
consent dismiss
- Paginate by clicking next (offset= deep-link was the page-2 tell), offset
URL kept as fallback
- Viewport jitter, mouse movement, longer scroll, selector retry on lazy load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- matrix response gains last_scraped per date (max scraped_at across all
hotels), so column headers show the newest scrape touching the date even
when the visible hotels were on a failed page
- cells >1h older than the column's latest scrape render italic with *
- every cell tooltip now includes the datestamp the price was scraped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A page-load timeout was logged and skipped, so a scrape could 'succeed'
with only page 1 of results — and the not_listed flagging then marked
every page-2 hotel absent, suppressing their last known rates.
- ScraperResult now tracks pages_requested/pages_ok
- scrape_date skips not_listed flagging when pages failed or when the
scrape saw <60% of the date's 7-day coverage baseline; scraped rates
are still saved, unseen hotels keep last known rate + scrape time
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Scrape queue generalised to date-range jobs; range and per-date scrapes
share one sequential queue (timeout scales with range length)
- Matrix refetches every 30s while a scrape runs so columns fill in live
- Eye icon per date header opens that night's location search on Booking.com
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- /analysis/hotel/{id}: lead_bucket alias in ORDER BY CASE broke Postgres; replaced
bucketed curve with per-days_ahead curve and reshaped response to the frontend
HotelAnalysis interface (strategy/advance_curve/dow_breakdown/sold_out_pattern)
- /analysis/hotel/{id}/timeline: accept ?date= (was rate_date, 422) and return
flat TimelineEntry array
- /analysis/hotels: alias to hotel_id/hotel_name/date_count for the selector
- strategy pcts default 0 (frontend calls .toFixed), added peak_months
- Market View badge now shows +/-% vs our rate instead of 100-index
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Location-search results aren't a fixed hotel set — a sold-out hotel drops
out and its last 'available' rate would remain the latest row for that
date, reading as a live price and skewing market averages. On each
successful per-date scrape, insert a NULL-rate 'not_listed' row for every
active hotel missing from the results (skipped if the parse found nothing,
which indicates scraper fault not absence). Failed/blocked scrapes write
nothing, so genuinely-stale data remains distinguishable by scraped_at.
Also: fix the DOW analysis to pick latest-then-filter so a not_listed
latest row drops the date instead of resurfacing an older rate, and widen
booking_com_id to VARCHAR(255) (some Booking slugs exceed 50 chars).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scraped room descriptions exceed VARCHAR(100) (StringDataRightTruncation),
and the failed insert poisoned the transaction so every subsequent save in
the batch died with InFailedSqlTransaction — scrapes reported success with
0 rows saved. Widen the column (with ALTER for existing tables) and roll
back after a failed save.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bookability showed no availability because newbook_occupancy_report_data
was never populated: add reports_occupancy client method and
sync_occupancy job, run before rates in both Sync Now and the daily
schedule (single fast API call)
- Category order: default display_order to the Newbook category id on
sync (was 0 → alphabetical), preserve manual order on re-sync, extend
PATCH to accept display_order, add up/down reorder arrows in Settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
run_fetch_current_rates is async; run_in_executor called it in a thread
and discarded the coroutine, so the daily scheduled sync silently did
nothing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- schema defined gross_rate/net_rate but every query (fetch job,
bookability, competitors) uses rate_gross/rate_net — rename the
columns, with an idempotent DO-block migration for existing tables
- roll back the session when a date fails so one bad statement no
longer poisons the whole sync run (every subsequent write was dying
with 'current transaction is aborted')
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rates Newbook instance returns categories as category_id/category_name
on each site; also skip inactive categories.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- require_cap was a Depends-factory but every call site uses it inline;
make it an inline checker (fixes 500 on /analysis/hotels, /direct/*)
- /analysis/comparison returned a per-date matrix the frontend never read;
return per-hotel aggregates (our/their avg, price index) and default to
all active competitors so the Market Comparison table works without params
- Room categories were never populated (lost in port): add sites_list fetch
to the Newbook client, categories list/sync/toggle endpoints, and a
Settings card — without included categories every rates sync exits early
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- RateAnalysis: null guard on price_incl in timeline trace
- Layout: explicit null check on alertCount, use ?? for user.name
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
