assert-what-the-generator-found
A generator that crashes tells you it failed; one that emits plausible output tells you nothing. Test what it found, never that it ran.
Conferred by a reviewer who read the card, not claimed by the file about itself. A card the agent distils during a run that consumed untrusted content is born tainted and is held for review before it is ever retrieved.
When it comes to mind
- wrote a code generator
- testing a schema dump
- the output looks fine
- extracting structure from a library
Avoid and Check are where the value usually is. Do is the section everybody writes.
Trigger
You have written something that reads one representation and emits another: a schema dump, a reference extractor, a migration, a scraper, an index builder. Anything whose output nobody reads in full because there is too much of it.
It does not apply to a function returning one value you can eyeball. The risk here is specific to output that is large enough that "it looks right" is the only review it will ever get.
Do
- Before writing the test, name the count. How many commands, rows, files, or fields should come out? Get that number from somewhere other than the generator — the source, the docs, a manual count.
- Assert the count, or a floor on it.
assert len(groups) >= 10, notassert result. - Assert the presence of specific, named items you know must exist. Three or four is enough, and pick ones from different shapes of input.
- If the generator classifies things, assert that each class is non-empty. A classifier that puts everything in one bucket is the failure this catches.
Avoid
assert build() — it passes when the generator returns an empty list, a partial list, or a list
where every item is subtly the wrong kind.
Also avoid trusting isinstance when walking a third-party structure. Libraries vendor their
dependencies: a TyperGroup is not an instance of the click.Group your file imported, because
Typer ships its own copy of Click. Ask whether the object has the thing you need — a commands
mapping, an items method — rather than what class it claims to be. Duck-typing survives a
vendored dependency and a major version bump; a class check survives neither, and fails by
misclassifying rather than by raising.
Check
Break the generator on purpose and watch the test fail. Comment out the recursion into subcommands, or make the type check reject everything, and run the suite.
If the suite still passes, the test asserts that the generator ran, and you have written the test that this skill exists to prevent.
Risk
Hard-coding an exact count makes the test a maintenance chore: every legitimately added command
turns it red. Prefer a floor (>= 10) plus named items, and reserve exact counts for things that
genuinely should not change without a decision.
There is also a limit. These assertions catch a generator that dropped a whole category. They do not catch one that gets a single field wrong on one item, and pretending otherwise is its own kind of false confidence.
Use it
The card is data. Clone the repository and import it by path — anything that arrives over the network is treated as tainted and held for approval, which is the behaviour you want and the reason there is no one-line installer here.
git clone https://github.com/brcampidelli/chimera-agent.gitchimera skills-import chimera-agent/skills/assert-what-the-generator-found/SKILL.mdIntegrity
SHA-256 of the file as published. An importer can check that what it received is what this page showed.
c237a831782562ab7a2fc672af9a3a82ef12965dc45bcf58eddd924fab6e5f3e