Your agent pages through a public API to pull a product catalog or a list of organizations. The script exits cleanly and the output has more than ten thousand rows. The agent reports "collection complete." Yet those rows may be the same hundred rows repeated, only the first part of the list, or half "quota exceeded" responses.
Why a crawl that exits cleanly may not have everything
A paginated crawl can collect too little, collect the same thing over and over, or count failures as done — and still exit normally. "It finished" and "it got everything" are separate facts, and only the second needs reconciling against the source's own total. A crawler cannot tell by itself whether a response is truly the next page or whether the list really ended there. So completeness is checked outside the crawler, with numbers.
Three ways a crawler quietly comes up short
1. The API ignores the page number and returns the same page. Some APIs, in a keyless sample or trial mode, ignore the page parameter and always return page one. No error code, same row count, so the crawler believes it got a hundred good rows per page.
2. It stops before reaching the end. When automation reads a screen, a list that lazy-loads on scroll contains only what is visible at first, so a count of it comes out far too low. With APIs, an end condition like "stop on an empty page" can cut the crawl at a transient empty response in the middle.
3. Failures get counted as received. A resumable crawler that skips keys it already has is convenient, but if a quota response (HTTP 429) is written down as a failed row, that key counts as "received" and the next run never retries it. And if last cycle's output file is still there at the start of a new cycle, every key looks "already done" and the run collects zero rows.
What to check before collecting
Two checks that take a minute, before the real run:
- Is the first row of page 2 different from the first row of page 1? If not, paging is not working. Get an API key, use the source's bulk download, or split by filters like region or category and fetch each piece small
- Does the source tell you its total? A
totalCount-style field, an admin API listing, a number on the portal. Without a baseline you have no grounds to say "everything." If there is an API, count with the API rather than a screen
What to reconcile after collecting
Before reporting "done," write down three numbers.
- Source total = rows received? If not, find where it got cut
- Unique keys = rows? Far fewer unique keys means you fetched the same page repeatedly
- Failed rows = 0? Never count a failed row as done; leave it for the next run to retry
For resumable crawlers, add a few more rules:
- A new cycle moves last cycle's output aside and starts fresh (
--fresh) - After N consecutive quota responses, treat the quota as exhausted and stop. Hammering on only piles up failed rows
- The loading step (into your database) skips failed rows and never overwrites existing values with empty ones
If an agent does this work, you can hand it this instruction:
When collecting a paginated list:
- Before the full run, fetch pages 1 and 2 and confirm their first rows differ. If they match, stop and report
- Record the total the source declares first. If no total is available, say so in the report
- When done, report four numbers: [source total / rows received / unique keys / failed rows]
- If the four numbers do not reconcile, do not report "done"
What to automate when you cannot recollect everything
If a full recollection takes hours and cannot run every time, automate the verdict, not the collection. Periodically compare a random sample of a few dozen items with the source, record the mismatch rate, and alert when it crosses a threshold. Record each time whether the alert path itself is alive, so you never discover it was dead only in the week something broke.
What happened to us
- Repeated pages: collecting about 12,000 schools from a public education data API without a key, we found page 2 was identical to page 1. No error code, no change in row count.
- Lazy-loaded list: reading an analytics admin screen with automation to count registered properties gave 6; the real number was 25. The list had not been scrolled to the end; counting through the admin API fixed it.
- Failures counted as done: a resumable crawler that recollects about 12,000 organizations monthly from a public API either counted quota responses as received or finished with zero rows because of last month's file. We stopped counting failed rows as done, halted on consecutive 429s, and made each cycle start fresh.
- Sample verdict: on the same service, a weekly check comparing 80 sampled items with the source caught the first threshold breach (25% mismatch against an 8% threshold).
In every case the crawler ended in success. What told us it had come up short was not its exit status but numbers counted from outside.
PreviousSearch engines and analytics count URLs and status codes, not screens — design the signal you send to machines separately