Flag hotels absent from successful scrapes as 'not_listed' + widen booking_com_id
Location-search results aren't a fixed hotel set — a sold-out hotel drops out and its last 'available' rate would remain the latest row for that date, reading as a live price and skewing market averages. On each successful per-date scrape, insert a NULL-rate 'not_listed' row for every active hotel missing from the results (skipped if the parse found nothing, which indicates scraper fault not absence). Failed/blocked scrapes write nothing, so genuinely-stale data remains distinguishable by scraped_at. Also: fix the DOW analysis to pick latest-then-filter so a not_listed latest row drops the date instead of resurfacing an older rate, and widen booking_com_id to VARCHAR(255) (some Booking slugs exceed 50 chars). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
75c2adf803
commit
ba6c000903
3 changed files with 31 additions and 5 deletions
|
|
@ -109,15 +109,17 @@ async def analyse_hotel(
|
|||
ROUND(AVG(rate_gross)::numeric, 2) AS avg_rate,
|
||||
COUNT(DISTINCT rate_date) AS date_count
|
||||
FROM (
|
||||
-- Latest row per date first, THEN filter: a 'not_listed'
|
||||
-- latest row must drop the date, not resurface an older rate
|
||||
SELECT DISTINCT ON (rate_date)
|
||||
rate_date, rate_gross
|
||||
rate_date, rate_gross, availability_status
|
||||
FROM booking_com_rates
|
||||
WHERE hotel_id = :hid
|
||||
AND rate_date BETWEEN :from_date AND :to_date
|
||||
AND availability_status = 'available'
|
||||
AND rate_gross IS NOT NULL
|
||||
ORDER BY rate_date, scraped_at DESC
|
||||
) latest
|
||||
WHERE availability_status = 'available'
|
||||
AND rate_gross IS NOT NULL
|
||||
GROUP BY dow, dow_label
|
||||
ORDER BY dow
|
||||
"""),
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue