Scraper: don't flag not_listed from partial scrapes
A page-load timeout was logged and skipped, so a scrape could 'succeed' with only page 1 of results — and the not_listed flagging then marked every page-2 hotel absent, suppressing their last known rates. - ScraperResult now tracks pages_requested/pages_ok - scrape_date skips not_listed flagging when pages failed or when the scrape saw <60% of the date's 7-day coverage baseline; scraped rates are still saved, unseen hotels keep last known rate + scrape time Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
de8c4ec257
commit
218b2f45f6
3 changed files with 62 additions and 6 deletions
|
|
@ -59,6 +59,8 @@ class ScraperResult:
|
|||
rates: List[RateData] = field(default_factory=list)
|
||||
error_message: Optional[str] = None
|
||||
page_content_sample: Optional[str] = None # For debugging
|
||||
pages_requested: int = 0 # Result pages we set out to fetch
|
||||
pages_ok: int = 0 # Pages that loaded and parsed cleanly
|
||||
|
||||
|
||||
class ScraperBackend(ABC):
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue