Silent scraper breakage (200-with-empty-body) went unnoticed for 16 days
because nothing flagged it. Now, after each daily direct scrape, a health
check compares every enabled competitor's freshest captured rate against a
staleness threshold (default 5 days, config: scrape_health_stale_days).
State lives in a new direct_scrape_health table so we alert on transitions
only — one email when a competitor goes stale, one when it recovers, never a
daily repeat. Email uses the stack's shared SMTP integration via
central_settings (recipient: scrape_health_alert_email, else SMTP reply_to/
from) — no new mail secrets in this app. The current stale set also drives an
in-app warning banner on the Direct Rates page (GET /direct/health), with an
on-demand re-check endpoint (POST /direct/health/check).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both competitor booking engines changed their API contracts, silently
breaking their direct scrapes ~16 days ago (200 responses with empty
bodies, so no exception was ever raised):
- Guestline: the /enhanced availability endpoint now returns a bare []
for every date. Switch to the base /api/availabilities/{coll}/{hotel}
endpoint, which returns {"rooms":[...]} with the same room shape.
- Mews: getAvailability now rejects the old body with "Invalid
EnterpriseId" — it requires enterpriseId + serviceId. getPricing
additionally requires currencyCode. Capture the enterprise's
defaultCurrencyCode in _ensure_config and send all three.
Also stop last_scraped_at advancing when a run saves 0 rows, so the
overview no longer reports a phantom-fresh scrape while the per-date
data stays stale — the symptom that surfaced this.
Verified live: Three Ways now yields 41 rows/night, Old Stocks 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Intraday rescrape jobs are distributed evenly between the main nightly run
(05:20) and cover only the next 30 days — lightweight complement to the
full 720-day nightly sweep. Interval is configurable from the Newbook tab
in Settings and takes effect immediately without a container restart.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
get_medium_priority_dates and get_low_priority_dates both require a
SyncSession argument; the endpoint was calling them without one, causing
a TypeError → 500 on every load of the schedule card.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parity logic:
- Extract _fetch_latest_alerts() and _upsert_alerts() helpers so the alert
upsert loop is no longer duplicated between the daily job and per-date runs
- Add run_parity_check_for_date(date) which runs the full comparison +
alert upsert for a single date
Scraper integration:
- _safe_parity_recheck(date) wrapper (never raises) called after each
successful date in both search-results and hotel-page workers; hotel-page
mode waits until all hotels for the date are done before rechecking
API:
- POST /competitors/parity/check-date?rate_date=YYYY-MM-DD for manual recheck
Frontend:
- Recheck button on every parity alert row (all statuses); invalidates
alerts list and badge count on success
- parity-alert-count badge polls every 15s (was 60s) so new alerts from
the scheduled job or post-scrape rechecks appear quickly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scrape_batch_id in booking_com_rates is a uuid column, not text.
Casting bsl.batch_id::text made the comparison uuid = text which
Postgres rejects. Drop the cast — both sides are uuid.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend: POST /competitors/scrape/scheduled triggers run_scheduled_booking_scrape
in a background task — repopulates the queue with today's high/medium/low
priority dates and processes it. Returns 400 if scraper is disabled, 409 if
already running.
Frontend: button in the Automatic Schedule card to trigger the scheduled job
manually (useful after an interrupted overnight run). Manual Scrape description
now shows which backend mode is active (hotel-page vs search-results) so it's
clear what the date range scrape actually does.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend: atomic += increments to hotels_found, rates_scraped,
dates_completed, dates_failed after each date in both worker types
(search-results and hotel-page). Safe for concurrent workers because
PostgreSQL evaluates x = x + delta atomically per statement.
If the container is killed mid-run, the counts already reflect what
was done rather than zeroing out.
Frontend: new Progress column showing X/Y (NN%) of dates completed;
highlighted in gold while the batch is running. scrape-history query
auto-refetches every 5s whenever a running entry is present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cleanup_stale_batches and force_reset_scraper were leaving hotels_found
and rates_scraped at 0 because update_scrape_batch never ran — the
container was killed before it could. Now we subquery booking_com_rates
by scrape_batch_id to show what was actually saved before the interruption.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- DISTINCT ON tiebreaker was undefined when scraper writes multiple room
types in one batch (same scraped_at); adding rate_gross ASC ensures we
always pick the cheapest (lead-in / best available) rate, matching like
for like against the Newbook BAR tariff
- Past-date active alerts were never touched (start = today meant they
fell outside the query window); now resolved at the top of each run
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SQLAlchemy's text() parser treated the ':' in '::date' (PostgreSQL cast syntax)
as a second unbound parameter after :start/:end, producing malformed SQL and
crashing the scheduled Booking.com scrape silently with no history entry written.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Original scrapy approach: extract <li> condition lines from each rate plan row
and text-search for known keywords. No match = null (unknown), not assumed false.
Adds breakfast_text, cancel_text, payment_text columns to booking_com_rates.
Booleans now nullable (null = not mentioned, true/false = explicit signal).
JS extraction searches all <li> items in the row (falls back to newline-split
innerText if none). Breakfast: 'breakfast' keyword. Cancel: 'free cancellation',
'non-refundable', 'total cost to cancel', 'fully chargeable'. Payment: 'no
prepayment', 'pay at the property', 'pay online'. data-fltrs used as fallback.
API snapshot endpoint now returns breakfast/cancel/payment text strings. Old
boolean-only rows degrade gracefully to derived labels.
Modal plan rows replace the fixed meal|cancel|price column layout with a single
stacked conditions cell: 0-3 lines depending on what the page actually shows.
Breakfast green when 'included', cancel green when 'Free cancellation…'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Breakfast was keyed off cells[2] specifically; hotel page templates vary and
the conditions column can appear at a different index, causing breakfast_included
to always be false for own/competitor rates. Now scans full row innerText and
also checks data-fltrs.mealplan/breakfast_included as a secondary signal.
Free cancellation extraction similarly updated to search all cells rather than
assuming a fixed column.
Also ships the update-available banner (polls /health every 2 min, prompts
reload when the version hash changes after a deploy).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three read-only endpoints under /reporting/:
- /hotels hotel dimension table (tier, stars, review score)
- /rates full rates fact — all sources, room types, scrape history in one flat table
- /occupancy Newbook occupancy per date × room category
Rates UNION covers Booking.com scrapes, direct competitor engines (with
configured room/rate labels), and own hotel Newbook headline rates.
Authenticated via X-API-Key header; key stored in system_config.reporting_api_key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Medium and low priority tiers were generating full static date ranges
(150 and 185 dates) every day. With a queue limit of 200, high (31) +
medium (150) consumed the entire budget, leaving only ~19 slots for low
priority — causing the observed ~6 month cap.
Medium now selects the 60 oldest-scraped (or never-scraped) dates from
the days 31-180 window; low selects the 30 oldest from days 181-365.
Per-run budget drops from ~365 to ~121 dates, and coverage naturally
cycles through the full year: medium every ~2-3 days, low every ~6-7 days.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema: migrate newbook_occupancy_report_data from single-row upsert to
snapshot model (drop unique constraint, add valid_from / last_verified_at)
matching the pattern used by newbook_current_rates.
Backend: sync_occupancy now inserts a new row only when occupied/available/
maintenance figures change, otherwise bumps last_verified_at. New endpoint
GET /bookability/occupancy-history/{category_id}/{date} returns the timeline.
Rate matrix query updated to DISTINCT ON for the multi-row table.
Frontend: clicking any cell in the Bookability matrix opens a modal with
two stacked Plotly charts — rate history per tariff (step lines, green/red
markers for available/unavailable) and occupancy pick-up over time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Eviivo: no-dates page fetch returns all room types with data-item-name and
data-rate-plan-name; single request populates the full name catalogue
- QBook: item_name already in /api/pull response; extracted on a dummy date
call; no distinct rate plan names exposed so rate_labels left empty
- Mews: getCalendarData (bookingEngineId field) returns resourceCategories[].name
and rates[].name as {en-US: ...} dicts; both room and rate labels populated
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A new toolbar row on the Rate Matrix lets users pick a reference datetime
(defaults to 24h ago, with 24h/3d/7d presets) and see ▲/▼ triangles
next to each competitor's BAR where the rate has moved ≥50p since then.
Two backend endpoints:
- GET /competitors/rate-changes?since= — static datetime comparison
- GET /competitors/rate-changes-vs-own — dynamic: diffs against the
timestamp our own Newbook rate last changed per date (highlights
competitor moves made in response to our own pricing decisions)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Some DataImpulse residential IPs drop Chromium's TLS 1.3 GREASE/key_share
extensions mid-handshake (EOF error), while TLS 1.2 negotiation succeeds.
Booking.com supports TLS 1.2 so this should not affect scrape results.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse has two broken gateway nodes (.89, .97) and one working (.105).
Production DNS round-robin consistently landed on broken nodes. Fix is to
pin the proxy host to 67.213.121.105 in system_config. Revert to Chromium.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Booking.com detects headless Chromium via TLS ClientHello fingerprint (JA3)
and immediately flags the residential proxy IP on first contact. Firefox has
a different TLS fingerprint and GREASE extensions that aren't yet flagged.
Image blocking via Firefox preference (permissions.default.image=2) replaces
the Chrome --blink-settings flag. Dockerfile updated to install both browsers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
context.route() (request interception) breaks URL-embedded proxy auth in
Chromium — DataImpulse proactive CONNECT auth stops working when interception
is active. Replace route handler with --blink-settings=imagesEnabled=false
Chrome flag to block images without touching the interception layer.
Restore DataImpulse URL-embedded credentials (unchanged from before IPRoyal work).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
URL-embedded proxy credentials are dropped by Chromium when request
interception (context.route) is active. Both providers now use separate
username/password fields; DataImpulse incurs a 14s 407 round-trip on
context rotation but this only happens every 40 requests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse username contains semicolons (;sessid.ID) that weren't being
URL-encoded in httpx_proxy_url, causing silent parse failures. Also use
httpx.Proxy object instead of raw string (consistent with httpx_proxy()),
and capture repr(e) so empty-message exceptions show their type.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Separate verified_at (last checked) from valid_from (last rate change)
in the matrix response. Column headers show when rates were last checked;
hovering shows both "Checked: X" and "Changed: Y" so users can distinguish
a manual refresh that verified unchanged rates from one that found new prices.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cf.bstatic.com (images + CSS CDN) was consuming 1.5 GB per scrape run.
Added route interception to abort image, stylesheet, font, and media
resources, plus pure tracking domains. The DOM extraction only reads
HTML attributes — no visual resources are needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch date_last_updated from valid_from (rate change time) to
last_verified_at (last check time). All dates verified in the same daily
run now show a consistent timestamp rather than varying by when rates
last changed. Also commit per-date instead of batching 10 days — prevents
a single API error from rolling back up to 9 preceding committed dates.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
URL-embedded credentials are silently dropped by Chromium for IPRoyal
(proxy auth never sent → every page.goto times out). Separate username/
password fields work correctly — IPRoyal responds to the 407 challenge
quickly so there is no latency penalty unlike DataImpulse (~14s).
DataImpulse keeps URL-embedded credentials to avoid that round-trip.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auto-detect provider from hostname. DataImpulse appends session/country to
the username (LOGIN__cr.gb;sessid.ID); IPRoyal appends them to the password
(PASS_country-gb_session-ID_lifetime-30m). Both providers use the same
host/port/username/password/country config fields in Settings.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DataImpulse endpoints are intermittently broken — some return EOF after
CONNECT immediately, others work fine. DNS round-robins between them so
the same request can fail 3× then succeed on the 4th. Timeouts were
returning blocked=False and silently failing with no retry. Now treated
as retryable blocks (with context rotation) so a fresh endpoint is tried.
Max retries 2→4 (5 total attempts) to give enough chances to land a
working DataImpulse endpoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Header-based auth made no difference; the real issue is the hotel network
firewall blocking HTTPS CONNECT tunnels on port 823. Reverted to URL-embedded
credentials (original approach). Fix requires DataImpulse to enable port 443.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Embedding credentials in the proxy URL (http://user:pass@host:port) was
breaking HTTPS CONNECT tunnels on the hotel network. Switching to separate
username/password fields (Playwright) and httpx.Proxy(auth=...) sends a
Proxy-Authorization header instead, which passes through correctly.
With DataImpulse IP whitelisting the 407 round-trip is skipped anyway so
there is no latency penalty.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
asyncio.gather iterated undefined `shards` — should be `date_shards`
(variable renamed when switching from hotel-first to date-first sharding).
Caused every hotel-page scrape to immediately fail with NameError.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Workers now own a slice of dates and scrape all hotels per date before
advancing. A block or interruption leaves complete dates rather than
some hotels fully done and others not started.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old Stocks block_id has 0 in position 2 (not the persons count) so all
its rates were stored as max_persons=0 and deprioritised in the matrix.
Now reads max_persons from the visible 'Max persons: N' span (matching
the original scrapy spider), with aria-label/title fallback. block_id
segment 2 is unreliable across different hotel layouts.
Also treat max_persons=0 as unknown in the matrix ORDER BY priority.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add UA rotation (8 real Chrome UAs), random viewport, en-GB locale,
Europe/London timezone, stealth browser args to suppress automation
signals, and a per-context init script masking navigator.webdriver,
plugins, and chrome runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous approach walked siblings from [id^="room_type_id_"] anchors
and only processed rows with js-rt-block-row class, causing the first rate
row of some room types to be silently skipped when that class was absent.
Now iterates all #available_rooms tbody tr rows with data-block-id (same
strategy as the original scrapy spider), using the room_type_id_ element
presence to identify room names only on the first row, with fallback to the
stored name for subsequent rows of the same room type.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Include tier=own in get_active_hotels() so the own hotel Booking.com
listing is scraped alongside competitors. Change matrix max_persons
filter from hard WHERE to ORDER BY priority so hotels with only
1-person rates (e.g. Old Stocks) still show up rather than being
silently excluded.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_proxy_enabled() was checking cfg.get('server') but load_config() returns
{host, port, username, ...} — no 'server' key. Server URL is built by
proxy_util.playwright_proxy(). This meant proxy was silently disabled
and _proxy_kwargs() would also KeyError if called.
- _proxy_enabled() now delegates to proxy_util.is_enabled() (checks host+username)
- _proxy_kwargs() now calls proxy_util.playwright_proxy() with credentials
embedded in the URL (avoids 14s DataImpulse 407 round-trip)
- _get_context() generates a sticky session_id per context so each worker
gets a distinct residential IP lane
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The proxy-off → serial guard was correct for search-results (all workers
hit the same URL), but hotel-page workers each scrape a different hotel's
property page — parallel requests look like multi-tab browsing, not a
hammered aggregation endpoint. Added require_proxy param to
_effective_concurrency; hotel-page path passes require_proxy=False.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: restrict hotel-page scraper to 'competitor' tier only (was own+competitor); own hotel rates come from Newbook API, not Booking.com
- Backend: POST /competitors/discover — runs search-results scrape for one date to find market hotels regardless of current backend setting
- Backend: POST /competitors/hotels — add hotel manually from Booking.com URL + name + tier; upserts on slug conflict
- Frontend (Settings tab): Discover Market Hotels button added below manual date-range scrape
- Frontend (Hotels tab): Add Hotel form at top — paste URL (auto-derives name from slug), choose tier, submit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Market-tier hotels were auto-discovered from search results and don't
need room-level rate tracking — scraping all 25+ of them per date was
unnecessary. Only 'own' and 'competitor' hotels are now scraped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: extract max_persons from block_id (3rd segment) instead of cell text; cell[0] was picking up the price value (e.g. 189/166) causing the max_persons=2 filter to exclude all scraped rows and show "No rates available" in snapshot
- MarketView: add namelength: -1 to Plotly legend to prevent label truncation; increase bottom margin from 50→65 and legend y from -0.25→-0.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 'We have X left' scarcity indicator only appears when availability is
low (typically ≤5). The quantity <select> dropdown in the booking form
always shows 0..N where N = actual available qty (capped at 10).
Extract max option value from the <select> in the last table cell per rate
plan row, falling back to regex parsing of the cell text if the select
isn't rendered, then to the scarcity text as a last resort.
Stored in both rooms_left and available_qty on RateData.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Modal now shows two sections:
- Current availability: all room types with rooms-left count, then each rate
plan (meal plan × cancel policy × price) from the latest scrape batch
- Rate history chart: one line per room type, cheapest 2-adult rate per
scrape run (max_persons filter added to exclude 1-adult variants)
New endpoint: GET /competitors/hotels/{id}/rate-snapshot/{date}
Returns all rate plan variants grouped by room type from the latest batch.
Handles legacy single-row data (pre hotel-page scraper) gracefully.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switches booking.com scraping from search-results page (Cloudflare-targeted)
to individual hotel property pages (not CF-protected). Each page load returns
all room types, all rate plan variants (room-only/B&B × refundable/non-ref ×
1-2 adults), and availability counts.
Key changes:
- New PlaywrightHotelPageBackend: proxy reuse until block, rotate on CF/WAF
- booking_scraper.py: _run_hotel_page_scrape(), scrape_hotel_date(),
_scrape_hotels_concurrent() — sharded by hotel so one proxy session covers
all dates for one hotel (looks human)
- schema.sql: ADD COLUMN rate_plan_id, max_persons on booking_com_rates
- competitors API: filter max_persons=2, order by rate_gross ASC as tiebreaker
so DISTINCT ON returns cheapest 2-adult rate from latest batch
Enable via Settings → Scraper Backend → playwright_hotel_page
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When page.goto() times out while Cloudflare's JS challenge redirect is in
progress, page.content() silently blocks until that navigation completes
before raising — adding ~1m46s of hidden delay per blocked-page attempt.
Fix: skip page.content() entirely when page_loaded=False (goto already
timed out), setting content="" so block detection treats it as no signal.
Also break out of the page loop early when page 1 failed to load with 0
hotels — no point spending another 60s on the offset-URL page 2 when the
browser is in a Cloudflare challenge state.
Per-blocked-attempt time: ~4 min → ~1 min (30s goto + 30s selectors).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Timeouts were set for the old proxy 407 round-trip world (14.5s latency
per challenge). Now that credentials are embedded in the proxy URL, pages
load in ~7s — the 90s goto timeout was forcing 5+ minutes of wasted wait
on Cloudflare-blocked dates before rotating sessions.
homepage warmup goto: 90s → 20s
page 1 search goto: 90s → 30s (4× normal load time headroom)
property-card waits: 30s → 15s (cards appear in <3s on clean pages)
Default concurrency 3 → 6: each worker uses its own residential proxy IP
so parallelism is safe. ~0.4 GB per Chromium; 6 workers fits in 4 GB LXC.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
