An AI writes posts for your blog, store or internal wiki every day, and it tags them too. The first few weeks look fine. A few months later you open the tag list and find dozens or hundreds of tags attached to a single post each — a neighbourhood restaurant, a specific earbud model, "March 2026 rentals". Every tag gets its own page, so the site keeps growing one-post listing pages, and search engines treat them as thin content: they skip indexing them or discount the site's overall quality. The usual quick fix is to put noindex on the tag pages.
Why do AI-generated tags scatter over time?
If you leave naming open to a generative model, the vocabulary grows in proportion to the number of posts. For each post the model picks the words that describe that post best, and the words that best describe a single post are usually its proper nouns — places, products, dates. Proper nouns don't repeat in the next post. So tag pages grow almost one-for-one with posts, and each stays a one-post page.
Concept tags move the other way. Concepts like "rentals", "job change" or "wireless earbuds" gather more posts per tag as content accumulates, so their pages get thicker. Both are "tags", but concept tags get better with time, and proper-noun tags get worse.
This is not just about tags. Categories, sentiment labels, support ticket types, product classes — any field where a model picks a name behaves the same way. An open vocabulary grows a longer tail as data piles up.
Doesn't noindex fix it?
noindex hides the symptom. It only keeps search engines from indexing the page; the thin pages keep being created, and internal links, sitemaps and crawl budget keep being spent on them. And they remain useless to people. Put the fix on the side that creates the tags, not the side that renders pages.
How do you close the vocabulary?
- Build a fixed list of concept terms. The existing tags with three or more posts, plus a few dozen concepts that cover the site's subject, are enough to start.
- Give the list in the prompt and have the model choose only from it. Not "add fitting tags" but "pick 1–3 from the list below".
- Let code handle anything outside the list. Map it through a predefined substitution table ("downtown restaurants" → "restaurants", a specific earbud model → "wireless earbuds"); if it isn't in the table, drop it. Do not ask the model to rewrite the post.
- Count what gets dropped. If the same word keeps getting dropped, the list is missing something. A person reviews it and adds it to the list or the table.
- Clean up the thin tags you already have. Re-tag existing posts through the table, and permanently redirect (301/308) each disappearing tag page to the closest concept tag, or let it 404.
Why not make the model rewrite when a tag is off-list?
A check that sends the whole output back for a rewrite on every format violation breaks more fields than it fixes. When the model rewrites, it also changes fields that were fine (body, summary), and those trip other rules. With a high violation rate, latency piles up and you eventually save empty fields. When code can fix a field reliably — and tags are exactly that — let code fix it. Keep rewrites for what code can't fix, and even then name the field and the reason, and take back only that field.
Lines for your agent or generation pipeline instructions:
Choose 1-3 tags (categories, labels) only from the list below. Do not invent new terms.
[list]
If a value outside the list appears, code maps it through the substitution table, or drops it if it is not there. Never regenerate the post because of this.
Log dropped values; each week a person reviews frequently dropped words and decides whether to add them to the list.
Checklist:
- Pull the posts-per-tag distribution. If the share of one-post tags rises over time, your vocabulary is open
- Check what share of one-post tags are proper nouns (places, products, dates)
- Does the generation prompt have a fixed list, and does code handle off-list values?
- Measure the regeneration rate on format violations. Above 20%, move that rule to code normalization or loosen it
What happened to us
- An information blog that an AI writes daily accumulated 86 thin tag pages. Most tags were place or product names attached to one post, and they grew with the post count. Instead of noindex, we gave the generator a fixed concept list and had code substitute or drop off-list tags.
- In another service, breaking a sentence-length rule (40 characters) in structured output triggered a full rewrite. The rewrite changed fields that had been fine, which tripped the rule again, and fields ended up saved empty. More than 20% of outputs tripped the rule, and each rewrite added about one second per turn (measured from response logs). We loosened the limit to 80 characters and changed rewrites to request only the failing field.