Ask three vendors what RLHF data costs and you'll get three proposals, two NDAs, and zero numbers. Human preference data is the most opaque line item in AI development — so here's the breakdown we wish someone had published for us.
What drives the price of preference data
RLHF-style work — ranking model responses, pairwise comparisons, detailed critique — prices on four variables:
1. Task complexity. "Which response is better?" with a rubric is one price. "Compare these two responses across helpfulness, accuracy, and tone, then write a 100-word justification" is another. Judgment time is the cost driver, and it scales fast.
2. Domain expertise. General-chat preferences need literate, careful generalists. Code review preferences need developers. Medical or legal preferences need credentialed reviewers, and the price reflects the labor market, not the platform.
3. Quality assurance depth. Single-pass labeling is cheapest; adding agreement scoring, golden questions, and expert review adds cost — but it's usually cheaper than re-labeling a bad dataset later.
4. Volume and turnaround. Rush jobs and tiny batches cost more per item. Steady volume at a sane deadline is where per-task prices get genuinely low.
Realistic per-task ranges in 2026
Based on what we see across the market and on our own platform:
- Simple pairwise ranking (general domain, clear rubric): roughly $1–4 per comparison on marketplace platforms; enterprise vendors quote multiples of that.
- Multi-dimensional ranking with written justification: roughly $3–8 per item, depending on rubric complexity.
- Domain-expert review (code, medical, legal, scientific): roughly $8–25+ per item, driven almost entirely by the expert labor rate.
On trAIn specifically, specialist tasks top out around $8 per task for complex RLHF critique, with simpler comparisons well under that — because you set the labeler rate directly and see exactly where the money goes.
The hidden costs that blow up budgets
The per-task number isn't where RLHF budgets die. These are:
Guideline drift. If your rubric is ambiguous, annotators diverge, agreement scores crater, and you re-label. Every hour spent tightening guidelines before launch saves days of re-work. A good platform pressure-tests your rubric on a pilot batch before scaling.
Throwing away signal. Many teams pay for rankings and discard the justifications — then pay again later for critique data. Collect richer feedback per task; the marginal cost is small and the data reuse is enormous.
Paying enterprise rates for startup volumes. If you're labeling 10k–100k comparisons rather than millions, enterprise minimums and platform fees can double your effective per-task cost. The premium is overhead, not quality.
A budgeting formula that works
Budget = tasks × per-task rate × 1.15 (pilot + rework buffer)
Run a paid pilot of 200–500 tasks first. Measure inter-annotator agreement on the pilot, fix your guidelines, then scale. Teams that skip the pilot routinely spend their rework buffer three times over.
The bottom line
RLHF data in 2026 should cost you single-digit dollars per item for general-domain work, full stop. If your quotes don't look like that, you're paying for someone's sales team.
Start a pilot on trAIn — send us your rubric and volume, and you'll get an actual per-task number back, usually within a day.