Scraper: don't flag not_listed from partial scrapes

A page-load timeout was logged and skipped, so a scrape could 'succeed'
with only page 1 of results — and the not_listed flagging then marked
every page-2 hotel absent, suppressing their last known rates.

- ScraperResult now tracks pages_requested/pages_ok
- scrape_date skips not_listed flagging when pages failed or when the
  scrape saw <60% of the date's 7-day coverage baseline; scraped rates
  are still saved, unseen hotels keep last known rate + scrape time

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jtricerolph 2026-07-05 15:29:06 +00:00
parent de8c4ec257
commit 218b2f45f6
3 changed files with 62 additions and 6 deletions

View file

@ -59,6 +59,8 @@ class ScraperResult:
rates: List[RateData] = field(default_factory=list)
error_message: Optional[str] = None
page_content_sample: Optional[str] = None # For debugging
pages_requested: int = 0 # Result pages we set out to fetch
pages_ok: int = 0 # Pages that loaded and parsed cleanly
class ScraperBackend(ABC):