Backend: atomic += increments to hotels_found, rates_scraped,
dates_completed, dates_failed after each date in both worker types
(search-results and hotel-page). Safe for concurrent workers because
PostgreSQL evaluates x = x + delta atomically per statement.
If the container is killed mid-run, the counts already reflect what
was done rather than zeroing out.
Frontend: new Progress column showing X/Y (NN%) of dates completed;
highlighted in gold while the batch is running. scrape-history query
auto-refetches every 5s whenever a running entry is present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Original scrapy approach: extract <li> condition lines from each rate plan row
and text-search for known keywords. No match = null (unknown), not assumed false.
Adds breakfast_text, cancel_text, payment_text columns to booking_com_rates.
Booleans now nullable (null = not mentioned, true/false = explicit signal).
JS extraction searches all <li> items in the row (falls back to newline-split
innerText if none). Breakfast: 'breakfast' keyword. Cancel: 'free cancellation',
'non-refundable', 'total cost to cancel', 'fully chargeable'. Payment: 'no
prepayment', 'pay at the property', 'pay online'. data-fltrs used as fallback.
API snapshot endpoint now returns breakfast/cancel/payment text strings. Old
boolean-only rows degrade gracefully to derived labels.
Modal plan rows replace the fixed meal|cancel|price column layout with a single
stacked conditions cell: 0-3 lines depending on what the page actually shows.
Breakfast green when 'included', cancel green when 'Free cancellation…'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Breakfast was keyed off cells[2] specifically; hotel page templates vary and
the conditions column can appear at a different index, causing breakfast_included
to always be false for own/competitor rates. Now scans full row innerText and
also checks data-fltrs.mealplan/breakfast_included as a secondary signal.
Free cancellation extraction similarly updated to search all cells rather than
assuming a fixed column.
Also ships the update-available banner (polls /health every 2 min, prompts
reload when the version hash changes after a deploy).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The management monitor checks http://{host}:{port}/health (no slug prefix).
Adds a location = /health alias alongside the existing /rates/health
so both the slug URL (browser/NPM) and the bare path (monitor) work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both Plotly charts in OccHistoryModal now share the same xaxis range,
computed as the union of valid_from timestamps across both datasets so
rate movements and occupancy pick-up align on the same time axis.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove shape:'hv' step interpolation from rate and occupancy traces —
defaults to diagonal lines matching the market view history graphs.
Also fixes CSS var() used as a Plotly colour (invalid, replaced with hex).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Apply 45% opacity to all column cells where stay_date < today —
date header, category fill, occupancy, and all tariff cells —
matching the same visual treatment as the Direct Rates table.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PWA exact-match location blocks had no root directive so try_files $uri
resolved against nginx's default /etc/nginx/html instead of
/usr/share/nginx/html. Moving root to server level fixes all three
PWA blocks (registerSW.js, sw.js, manifest.webmanifest).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Schema: migrate newbook_occupancy_report_data from single-row upsert to
snapshot model (drop unique constraint, add valid_from / last_verified_at)
matching the pattern used by newbook_current_rates.
Backend: sync_occupancy now inserts a new row only when occupied/available/
maintenance figures change, otherwise bumps last_verified_at. New endpoint
GET /bookability/occupancy-history/{category_id}/{date} returns the timeline.
Rate matrix query updated to DISTINCT ON for the multi-row table.
Frontend: clicking any cell in the Bookability matrix opens a modal with
two stacked Plotly charts — rate history per tariff (step lines, green/red
markers for available/unavailable) and occupancy pick-up over time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Apply 45% opacity to rows where stay_date < today so past dates are
visually distinguished while preserving the red/yellow/green availability
colour semantics. Expansion rows carry the same opacity.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A new toolbar row on the Rate Matrix lets users pick a reference datetime
(defaults to 24h ago, with 24h/3d/7d presets) and see ▲/▼ triangles
next to each competitor's BAR where the rate has moved ≥50p since then.
Two backend endpoints:
- GET /competitors/rate-changes?since= — static datetime comparison
- GET /competitors/rate-changes-vs-own — dynamic: diffs against the
timestamp our own Newbook rate last changed per date (highlights
competitor moves made in response to our own pricing decisions)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Separate verified_at (last checked) from valid_from (last rate change)
in the matrix response. Column headers show when rates were last checked;
hovering shows both "Checked: X" and "Changed: Y" so users can distinguish
a manual refresh that verified unchanged rates from one that found new prices.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switch date_last_updated from valid_from (rate change time) to
last_verified_at (last check time). All dates verified in the same daily
run now show a consistent timestamp rather than varying by when rates
last changed. Also commit per-date instead of batching 10 days — prevents
a single API error from rolling back up to 9 preceding committed dates.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scraping a past date via the hotel-page backend wouldn't return anything
useful (Booking.com shows no availability for dates gone by). The ↻
button is now only rendered for today and future dates.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: restrict hotel-page scraper to 'competitor' tier only (was own+competitor); own hotel rates come from Newbook API, not Booking.com
- Backend: POST /competitors/discover — runs search-results scrape for one date to find market hotels regardless of current backend setting
- Backend: POST /competitors/hotels — add hotel manually from Booking.com URL + name + tier; upserts on slug conflict
- Frontend (Settings tab): Discover Market Hotels button added below manual date-range scrape
- Frontend (Hotels tab): Add Hotel form at top — paste URL (auto-derives name from slug), choose tier, submit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Sold out + last price: show SOLD label above strikethrough price; remove price-index badge (rate no longer bookable, delta misleading)
- Sold out, no history: unchanged — shows 'Sold'
- Past days: price-index badge now grey/faded (#94a3b8 on #e2e8f0) instead of coloured — historical context only
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Scraper: extract max_persons from block_id (3rd segment) instead of cell text; cell[0] was picking up the price value (e.g. 189/166) causing the max_persons=2 filter to exclude all scraped rows and show "No rates available" in snapshot
- MarketView: add namelength: -1 to Plotly legend to prevent label truncation; increase bottom margin from 50→65 and legend y from -0.25→-0.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- MarketView: split tickformat '£,.0f' into tickprefix+'£' + tickformat ',.0f' (Plotly d3 format doesn't accept £ prefix inline)
- Settings System tab: add editable dropdown for booking_scraper_backend (hotel page vs search results) with description of the tradeoff; backend was previously read-only in the table
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Modal now shows two sections:
- Current availability: all room types with rooms-left count, then each rate
plan (meal plan × cancel policy × price) from the latest scrape batch
- Rate history chart: one line per room type, cheapest 2-adult rate per
scrape run (max_persons filter added to exclude 1-adult variants)
New endpoint: GET /competitors/hotels/{id}/rate-snapshot/{date}
Returns all rate plan variants grouped by room type from the latest batch.
Handles legacy single-row data (pre hotel-page scraper) gracefully.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the JWT expired, AuthGate redirected window.location to /auth/login
which loaded inside the portal iframe. The portal detected window.self !==
window.top and showed the EmbeddedFallback ("This app isn't available yet")
instead of the login page.
Redirect window.top instead so the portal itself navigates to /login.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: threading.Lock held indefinitely when Playwright browser
hangs inside run_in_executor (finally never fires from the async side).
Fixes:
- _acquire_scrape_lock/_release_scrape_lock track monotonic timestamp
- POST /competitors/scrape/reset force-releases the lock and marks any
running batch as interrupted (queue rows stay intact for retry)
- GET /competitors/status now includes lock_held_seconds
- APScheduler watchdog job every 30 min auto-releases if held >3h
- Settings → Scraper Proxy tab shows live lock status (green/amber)
with a Force Reset button requiring confirmation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sold-out cells: amber (was red), show last known available rate with
strikethrough. Past dates: grey, same strikethrough treatment, no eye
icon (Booking.com won't serve past availability), history chart still
accessible. Date column headers also hide the eye link for past dates.
Backend adds a second query returning last_available_rate (most recent
available + non-null rate_gross) per hotel+date for the full range.
Icons are now 12px and stacked below the rate text in a flex-column
cell layout. Price index badge sits between rate and icons.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace full-cell link with two small icon buttons in each matrix cell:
- Eye icon links to the hotel's Booking.com page for that check-in date
- LineChart icon opens a history modal (Plotly multi-line chart)
The history chart plots best available rate over scrape runs, with one
line per room type so that changes in which room is cheapest show as
separate traces rather than a single jumpy line.
Backend: GET /competitors/hotels/{id}/rate-history/{stay_date} groups
by scrape batch, takes MIN(rate_gross) per room type per run, returns
series [{room_type, points: [{t, rate}]}].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New GET /competitors/own-direct-rates: cheapest bookable non-dinner
Newbook tariff per date (same selection rules as the parity check).
'Show direct rates' now renders a 'Direct (Newbook)' sub-row under the
own-hotel row alongside the competitors' scraped direct rows; cell
tooltip names the tariff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
parity_match_mode config ('best_available' default | 'match_terms').
Best available = cheapest bookable non-dinner direct tariff vs the BC
lead-in (already BC's best available) — simplest like-for-like. Term
matching kept as an option for days where BC's cheapest basis differs
from direct's. Settings gains a comparison-basis select.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- config: parity_markup_value/_unit + parity_tolerance_value/_unit
(pct|gbp), legacy *_pct keys still read as fallback
- expected rate = newbook + £X or newbook × (1 + X%); breach test uses
the tolerance in its own unit
- Settings parity tab: unit selects + live worked example line
- Parity Alerts tab: unit-aware description, badge now shows £ deviation
alongside %
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Parity was read-only — the alerts table had no producer, so the Market
View badge could never fire. Now:
- jobs/check_rate_parity.py: daily 06:45 job comparing own Booking.com
lead-in rate vs cheapest Newbook rate per date, measured against an
EXPECTED markup (we deliberately price Booking.com higher to cover
commission): alert when deviation from newbook*(1+markup%) exceeds the
tolerance. Creates/updates active alerts, auto-resolves dates back in
line, leaves acknowledged dates alone.
- config keys: parity_check_enabled, parity_expected_markup_pct,
parity_tolerance_pct (system_config)
- POST /competitors/parity/check manual trigger; GET /parity now uses the
same markup/tolerance and cheapest-across-categories Newbook rate
- Settings -> Rate Parity tab: markup %, tolerance %, enable toggle,
run-now with result summary
- Market View -> Parity Alerts tab: status-filtered list w/ acknowledge
- Market View -> Hotels: direct-link dropdown per competitor (new PUT
/competitors/hotels/{id}/direct-link) — closes the never-written
direct_hotel_id gap so the matrix direct-rates sub-row can populate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- new GET /direct/hotels/{id}/config returns full hotel config plus
distinct room/rate IDs from scraped data (and any known via labels)
- Configure panel: benchmark room/rate selects, tier base room, per-room
£ offsets (base locked to 0), friendly room/rate names, room display
order (up/down) — saved via the existing PUT endpoint
- enables apples-to-apples estimated benchmark when only e.g. a suite
is left: bench = room_price - room_offset + bench_offset
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Direct-rates sub-row in Market View was dead: the frontend filters
competitors on direct_hotel_id but /matrix and /hotels never returned it.
Add direct_hotel_id to both queries (+ HotelResponse), and only render the
"Direct" sub-row when a hotel actually has a non-null direct rate (was
rendering all-dashes on an empty {} object).
Remove the pause-on-block flow entirely — dormant since rotate-on-block
replaced it (nothing set booking_scraper_paused=true after set_scraper_paused
was dropped): is_scraper_paused, /config/unpause, the /scrape paused guard,
ScraperStatusResponse.paused/pause_until, and the frontend Paused badge +
Unpause button. Trim now-unused datetime import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fan a scrape run out across N workers, each with its own DB session and
its own scraper backend — and since every backend picks a random sticky
session id, each worker scrapes from a distinct residential IP. Dates are
interleaved across workers so each covers a spread of the range. Cuts a
150-date batch from ~30-45 min to ~12-15 min at 3 workers.
- booking_scraper_concurrency config key (default 3); Settings → Scraper
Proxy has a "Parallel workers" field
- Forced to 1 when the proxy is off (N workers would share one IP and
hammer it) or when there's a single date
- Both manual and queue paths routed through _scrape_dates_concurrent;
workers mark their own queue items
- Per-worker rotate-on-block replaces the old global pause-on-block
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The location_search_url column existed but was dead — the scraper always
rebuilt the URL from the free-text location name. Now the Location
Configuration form takes a "Booking.com search URL" field: paste the
address-bar URL from a real search and the scraper lifts ss/dest_id/
dest_type from it (the most reliable destination pin). Falls back to
dest_id, then plain name. Server also extracts dest_id from the URL for
the column and derives a display name from ss when none is typed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "Scraper Proxy" tab: enable toggle, host/port/username/password/
country, Save, and a Test Connection button that reports the live exit
IP + country through the proxy
- Backend proxy config now lives in system_config (DB authoritative when
booking_proxy_enabled is set; BOOKING_PROXY_* env vars are the fallback)
- Dedicated /config/proxy GET/POST/test endpoints; password is write-only
(never returned, blank keeps the stored value) and masked in /config/system
- Surface proxy status keys in the read-only System tab
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Concurrent manual scrapes were interleaving (two Chromium sessions on one
LXC) causing the page timeouts behind partial results. SCRAPE_LOCK guards
run_manual_scrape and process_queue; the trigger endpoint returns 409 when
busy, and the frontend keeps the job queued and retries after 30s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- matrix response gains last_scraped per date (max scraped_at across all
hotels), so column headers show the newest scrape touching the date even
when the visible hotels were on a failed page
- cells >1h older than the column's latest scrape render italic with *
- every cell tooltip now includes the datestamp the price was scraped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Scrape queue generalised to date-range jobs; range and per-date scrapes
share one sequential queue (timeout scales with range length)
- Matrix refetches every 30s while a scrape runs so columns fill in live
- Eye icon per date header opens that night's location search on Booking.com
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- /analysis/hotel/{id}: lead_bucket alias in ORDER BY CASE broke Postgres; replaced
bucketed curve with per-days_ahead curve and reshaped response to the frontend
HotelAnalysis interface (strategy/advance_curve/dow_breakdown/sold_out_pattern)
- /analysis/hotel/{id}/timeline: accept ?date= (was rate_date, 422) and return
flat TimelineEntry array
- /analysis/hotels: alias to hotel_id/hotel_name/date_count for the selector
- strategy pcts default 0 (frontend calls .toFixed), added peak_months
- Market View badge now shows +/-% vs our rate instead of 100-index
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Bookability showed no availability because newbook_occupancy_report_data
was never populated: add reports_occupancy client method and
sync_occupancy job, run before rates in both Sync Now and the daily
schedule (single fast API call)
- Category order: default display_order to the Newbook category id on
sync (was 0 → alphabetical), preserve manual order on re-sync, extend
PATCH to accept display_order, add up/down reorder arrows in Settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- require_cap was a Depends-factory but every call site uses it inline;
make it an inline checker (fixes 500 on /analysis/hotels, /direct/*)
- /analysis/comparison returned a per-date matrix the frontend never read;
return per-hotel aggregates (our/their avg, price index) and default to
all active competitors so the Market Comparison table works without params
- Room categories were never populated (lost in port): add sites_list fetch
to the Newbook client, categories list/sync/toggle endpoints, and a
Settings card — without included categories every rates sync exits early
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
