The short answer: buy until your labeling spend would fund at least 5–10 full-time annotators plus a QA lead — somewhere around $300k–$500k per year — and you have steady, predictable volume. Below that, an in-house team's true cost (recruiting, management, idle time, QA tooling) almost always exceeds a platform's per-task pricing.
Here is the math and the reasoning, including the cases where building in-house is genuinely right.
The visible cost vs the real cost
The naive comparison is "platform price per task" against "annotator hourly wage." That comparison is wrong because in-house labeling carries costs that don't appear in the wage line:
- Recruiting and churn. Annotation has high turnover. Budget continuous recruiting as a fixed cost, not a one-time one.
- Management and QA. Roughly one QA/reviewer seat per 8–12 annotators, plus a lead who owns guidelines and edge cases.
- Tooling. You will need annotation interfaces, assignment logic, review queues, and export pipelines — built or bought (this is what platforms like Labelbox sell on its own).
- Idle capacity. Volume is bursty. You pay salaries through the troughs; a platform bills only for completed tasks.
- Ramp time. A platform starts this week. A hired team takes 1–3 months to recruit, train, and calibrate.
A defensible rule of thumb: multiply the raw wage cost by 1.6–2.0× to approximate loaded in-house cost before comparing against per-task pricing.
A worked example
Say you need 50,000 RLHF preference judgments this year.
- Platform route (trAIn): platform access from $30/month plus per-task rates published on the pricing page. Complex rating work runs around $0.81 per judgment, so 50,000 single-label judgments cost roughly $40,000 — with measured quality via golden questions, and no fixed overhead when the campaign ends. Want 3-label consensus? It roughly triples the per-item cost, which is why most teams run consensus on a sample and single-label the rest.
- In-house route: 50,000 judgments at ~1 minute each is ~830 hours — barely 0.4 of one annotator's year. But you cannot hire 0.4 of a calibrated team: recruiting two annotators, a fractional QA lead, tooling, and management time lands at $120k+ loaded, and that capacity sits idle between campaigns.
The crossover arrives when volume is both large and steady: at ~200,000 judgments a year the platform runs ≈ $162,000 (single-label) while a minimal in-house team — two annotators, fractional QA, tooling — loads to $250k+, so buying still wins but the gap narrows. Past 500,000 judgments a year ($400k of platform spend, the top of the breakeven range above), a steady in-house team's loaded cost becomes comparable and the control benefits start to tip the scale.
When in-house actually wins
Build the team when all of these are true:
- Volume is high and steady — enough to keep 5+ annotators busy year-round.
- The work is deeply proprietary — export-controlled, pre-announcement, or regulated data that cannot leave your perimeter.
- Labeling IS the product — e.g., you are a data vendor yourself, or the annotation judgment embeds irreplaceable domain expertise (clinical reads by your own clinicians).
- You need same-day iterative loops with labelers sitting inside your research team, multiple times per day, indefinitely.
Two out of four is not enough. The most common failure mode we see is a team that hires for peak volume and then pays full-time salaries through the valleys.
The hybrid most teams land on
The pragmatic middle: a small in-house group that writes guidelines, reviews samples, and handles the most sensitive 5–10% of items — while a platform absorbs the production volume. Your experts spend their time on rubric quality and audits, where their judgment is scarce, instead of on throughput.
Platforms with structured guidelines support (worked examples, edge cases, reference attachments) make this split work cleanly: your in-house team encodes the rubric once, and the workforce executes it with golden-question accuracy tracked continuously.
Try the "buy" side on a real task
The cheapest way to settle build-vs-buy is a pilot: take 500 real items, run them through trAIn with your actual rubric and a few golden questions, and compare measured accuracy and cost against your in-house estimate. See what the deliverable looks like — a sample export is downloadable there — and size the pilot from the public pricing tiers.