Training Models on Their Blind Spots

The failures that matter most in production are rarely the ones covered by an existing eval set — by definition, if we'd already written a test for it, we'd have caught it earlier. So we built a pipeline that goes looking for failures instead of waiting for them to get reported.

It clusters real conversations by outcome, surfaces the clusters that look like failure modes, and turns the clearest examples into labeled training data. The loop closes when the next model version is evaluated against the same clusters that found the problem in the first place.

The output is a training set that's shaped like our actual failure distribution, not like our assumptions about it.

(Placeholder post.)