Transforms that shrink their input silently drop content when the input breaks their assumptions — count what you meant to keep before and after

Your crawler collects tens of thousands of detail pages. To save storage it converts the HTML to plain text and strips every script tag. The crawl finishes without an error and the text looks fine. But the latitude/longitude fields you wanted for the map are empty on every row: the coordinates never lived in the page body, only inside the map initialization script. Or you simplify region boundary polygons with Douglas–Peucker before drawing them, and some regions vanish entirely while the map's fit-to-bounds lands somewhere odd. Again, not a single error.

Why do reducing transforms drop content without an error?

A transform that shrinks its input (text extraction, shape simplification, compression) does not fail when the input breaks its assumptions — it returns a plausible-looking result. The only way to see what went missing is to count what you meant to keep (field fill rate, point count, area, decompressed content) before and after, and compare.

Throwing things away is the whole job of a reducing transform, so "something was discarded" is never an alarm by itself. The trouble is that every such transform carries a hidden assumption about what is safe to throw away:

• HTML-to-text assumes meaningful values live in the visible body. A value that sits inside a script is classified as disposable
• Douglas–Peucker assumes a baseline from the first point to the last. On a closed ring the first and last points are the same, so the baseline has length zero, every point's distance computes as zero, and the whole ring collapses to two points
• Compression does not promise "same input, same bytes." The DEFLATE specification (RFC 1951) defines the compressed format, not which bytes a compressor must emit. A test that compares compressed output can disagree across machines even when the content is identical

In all three cases the transform succeeds. When an out-of-assumption input arrives, the shape of the result is still normal, and what went missing is invisible from the result alone. You only see it by comparing against the original.

What should you count?

Write down in one line what the transform must preserve, then measure it the same way on both sides. Mapped to common cases:

• Crawling and scraping: the fill rate of fields you must have (coordinates, prices, IDs). An online store that puts its product price only inside a JSON-LD script will lose the price when converted to text. A fill rate that drops from 100% to 0% means the transform discarded it
• Shape simplification: points remaining per ring (at least three) and area. Set a threshold for how much of the original area must remain. A polygon that became two points has zero area and trips the check immediately
• Compression and serialization: the decompressed content, not the compressed bytes. What you are protecting is the content, not the encoding

How do you handle inputs outside the assumption?

Counting tells you something went missing. To keep it from going missing, handle the out-of-assumption cases before the transform discards anything:

1. Extract before you discard. Ahead of the cleanup step that removes scripts, pull the values that live inside scripts (coordinates, IDs) with a parser or pattern and store them separately. Put the step that throws away the original last
2. Split inputs that break the assumption. Cut a closed ring in two at the point farthest from its start, simplify each half, and join them. Leave rings too small to split unsimplified
3. Run a small sample first. Before a full crawl or full conversion, run a few hundred items and check field fill rates and point counts. Finding empty fields after a multi-hour full run means running it again
4. Make tests guard content, not artifacts. A test that compares a committed compressed file with a freshly built one should decompress both and compare the content

When you hand this to an agent, write it down like this:

When adding or changing a transform that reduces its input (text extraction, shape simplification, compression, summarization):
1. State in one line what it must preserve (e.g. coordinate fields, points and area per ring, decompressed content).
2. Count that the same way before and after the transform. If it shrank, fail and stop.
3. Before any step that discards the original, extract values that exist only there (such as values inside scripts).
4. Find inputs that break the transform's assumptions (closed rings, zero-length baselines) and handle them separately.
5. Before a full run, check the counts from step 2 on a sample of a few hundred items.

What happened to us

• Crawling public disclosure pages (2026-10): while collecting tens of thousands of detail pages, scripts were stripped during HTML-to-text conversion, and the latitude/longitude that existed only in the map initialization code disappeared with no error. The fix extracts script coordinates before cleanup and checks coordinate fill rate on the first few hundred pages before the full crawl
• Map boundary shapes (2026-10): simplifying region boundaries with Douglas–Peucker collapsed closed rings (first point equals last) to two points, so the shapes vanished and the fit-to-bounds went wrong. Splitting each ring at the point farthest from its start, simplifying both halves and rejoining them brought the shapes back
• Checking baked compressed data (2026-10): a test that byte-compared a committed gzip file against the baking code's current output passed locally and failed only in CI, with the same input and the same compression level. Comparing the decompressed content instead made it pass regardless of machine

Checklist

• Did you write down in one line what this transform must preserve?
• Do you count it the same way before and after, and stop when it shrinks?
• Did you extract values that exist only in the original before discarding it?
• Did you find inputs that break the transform's assumptions and handle them separately?
• Did you check fill rates on a sample before the full run?