Flag hotels absent from successful scrapes as 'not_listed' + widen booking_com_id

Location-search results aren't a fixed hotel set — a sold-out hotel drops
out and its last 'available' rate would remain the latest row for that
date, reading as a live price and skewing market averages. On each
successful per-date scrape, insert a NULL-rate 'not_listed' row for every
active hotel missing from the results (skipped if the parse found nothing,
which indicates scraper fault not absence). Failed/blocked scrapes write
nothing, so genuinely-stale data remains distinguishable by scraped_at.

Also: fix the DOW analysis to pick latest-then-filter so a not_listed
latest row drops the date instead of resurfacing an older rate, and widen
booking_com_id to VARCHAR(255) (some Booking slugs exceed 50 chars).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-05 14:44:02 +00:00
parent 75c2adf803
commit ba6c000903
3 changed files with 31 additions and 5 deletions

View file

@ -109,15 +109,17 @@ async def analyse_hotel(
ROUND(AVG(rate_gross)::numeric, 2) AS avg_rate,
COUNT(DISTINCT rate_date) AS date_count
FROM (
-- Latest row per date first, THEN filter: a 'not_listed'
-- latest row must drop the date, not resurface an older rate
SELECT DISTINCT ON (rate_date)
rate_date, rate_gross
rate_date, rate_gross, availability_status
FROM booking_com_rates
WHERE hotel_id = :hid
AND rate_date BETWEEN :from_date AND :to_date
AND availability_status = 'available'
AND rate_gross IS NOT NULL
ORDER BY rate_date, scraped_at DESC
) latest
WHERE availability_status = 'available'
AND rate_gross IS NOT NULL
GROUP BY dow, dow_label
ORDER BY dow
"""),