Distilling Human Intelligence
Every lab we talk to wants to run more RLHF than it does. The thing stopping them is not money. It is that the machinery for collecting the data belongs to someone else.
So we built the machinery and hand it to the lab. The raters, the onboarding, the prompts and completions, the preference collection, the evals. It runs on your side, on the model you are training right now.
Why vendor data goes stale.
A batch takes weeks. By the time it comes back, the checkpoint it was sampled from is gone. You end up training a model on the preferences of a model that no longer exists.
Do that for long enough and the model learns the shape of the reward instead of the thing the reward stood for. You can see it in the output: rounded corners, neon light rails, the same four gestures in every answer.
The fix is to sample from the policy you have today. Almost nobody does, and the reason is simple. A vendor would need your weights, your training methods, and your compute. No lab is going to hand those over. Which means the collection has to move to you.
The loop, on your side.
- 1
You train
We sample from the checkpoint you have right now. Nothing gets frozen and mailed off for a month.
- 2
We generate
Prompts and completions come off that checkpoint, pointed at whatever behavior your researchers noticed this week.
- 3
We collect
Our raters grade them against your spec. We onboard them on it, and then keep onboarding them, because people drift away from a spec about as fast as you'd expect.
- 4
We audit
Evals run on fresh seeds. You find out whether the preference you just collected made it into the weights, and where the model is weak next.
Then it runs again, on the checkpoint that just came out of the last turn. The number we care about is how long it takes to get from a behavior somebody noticed to a dataset that fixes it. At most labs today that number is measured in months.
What you get.
- The raters
- People with the credential the campaign needs, filtered on skill, education, and how they scored before. They work inside your instance, not a vendor's.
- The onboarding
- Most bad annotation is really bad onboarding. So you get your spec, the examples that go with it, and a review layer that catches drift while it's still small.
- The generation
- Prompts and completions sampled off your own checkpoints, so the data matches the distribution you're actually training on.
- The evals
- The step nobody does today: proof that the preferences you paid for reached the weights, plus a list of what to fix next.
Nobody can prove the preferences landed.
The people who write the spec are not the people who check that the model learned it. In between sit the alignment team, the researchers, and HumanOps, each with a queue of their own. A preference goes in one end, and no one can point at the weights and say it arrived.
We run evals on fresh seeds after every turn. It is the evidence the person who wrote the spec would have asked for, except that nobody has to ask.
Talk to us.
Tell us what your RLHF stack looks like now and where it hurts. If you are training frontier models and the vendor loop is the thing slowing you down, we would like to hear about it.