Most products that put a model in front of users end up with two labels: AI-generated, and human-reviewed. I shipped that pair. It broke in a week, and not for a subtle reason.
The product is NearMetal, a job index for systems software engineers — kernel, compiler, database, storage, networking, virtualization, embedded, and performance roles. It reads company career pages directly, files each role into one of eight sub-domains, and links back to the employer’s own posting. No account, no résumé, nothing to buy. The only thing it really offers is that you can see how each row was classified.
The classifier is boring on purpose
There is no LLM in the classification path. It is a keyword and regex scorer: positive domain terms, language patterns, negative signals, and a hard disqualifier for titles that are not individual-contributor roles. A role needs a confidence of at least 0.55 and one matched domain before it counts as systems software at all.
The part that matters is that it scores each field separately. The sub-domain, the languages, the employment type, the seniority, the workplace type — five numbers, not one. A row can be obviously a kernel job and completely unclear about whether it is senior.
That is what forces the interesting decision. One score per row lets you pretend there is a single publish threshold. Five scores per row does not.
Two labels were a lie
The first schema had one column with two values: classified by the agent, or reviewed by a person. Then I did the obvious thing and let an agent work the review queue — re-reading the description, checking the sub-domain, flagging conflicts.
Now the column had nowhere to put the result. Writing it as human review is false; no person looked. Writing it as unreviewed throws away real work and leaves the queue permanently full, which means the queue stops meaning anything. Both options corrupt the one signal the product exists to provide.
So the column got four values instead of two: classified by the rule set, reviewed by an agent, agent-classified and human-confirmed, and entered by a person. On the public row that renders as three strings, in descending strength of claim:
Confirmed by a person 2026-08-04Reviewed by an agent 2026-08-04Classified by rule set
An agent review is never reported as a human confirmation. That is the whole rule, and it cost one migration.
What a guess is allowed to claim in public
The second decision was harder, and it is not “hide low confidence.”
Unverified claims about the employer stay hidden. If the scorer is unsure whether the role is Staff level, whether the remote scope is worldwide, or what the compensation is, NearMetal shows nothing rather than a hedge. A wrong salary is worse than a missing one, because the reader has no way to check it without leaving.
But NearMetal’s own sub-domain filing is always published, along with its confirmation state, even when confidence is low. That looks inconsistent until you notice the asymmetry: the filing is the row’s identity and its correction path. Hiding an uncertain classification does not protect the reader — it conceals the classification, so nobody can tell it is wrong. Showing it as unconfirmed invites the one-click report that fixes it. You can see both behaviors on the live index, and the rules themselves are published on the about page.
What this costs
It costs density. The index looks sparser than boards that guess at salary bands, and sparse reads as small.
It costs a label that sounds weaker. “Classified by rule set” is a worse marketing line than “AI-matched,” and it is the honest description of a regex scorer.
And the boundary band — the roles scoring between 0.55 and 0.79 — never fully clears. Those are exactly the rows where a human eye changes the answer, so the queue is permanent by design rather than a backlog to burn down.
The general version
Provenance is only worth displaying if the label carries the strongest claim in the chain and nothing more. The moment “reviewed” covers both a person and a model, the word costs the reader more than it gives them: they now have to assume the weaker meaning everywhere, including the places where you did the expensive thing.
Adding a fourth state took an afternoon. Deciding to add it was the actual work, because the two-state version was already shipped, already looked fine, and was already telling users something that was not true.