Skip to content

Leadership and readiness

Dataset operations capacity planning: 21% more intake, triple the queue

By SourceX Editorial · Reviewed by Noah Loul ·

11 sections

Short answer

Dataset operations capacity planning works stage by stage: arrivals after yield, touch hours by case type, rework, productive specialist hours and outside waits. The busiest specialist stage sets throughput, and its queue grows with utilization divided by one minus utilization. CRAN's public counts show the stakes: from early 2025 to early 2026, submissions rose 20.6 percent, hand-triggered review actions 46.5 percent, and its first-time queue tripled.

Checklist

Dataset operations capacity

This page covers dataset operations capacity planning: 21% more intake, triple the queue.Illustrative

Key takeaways

  • Model capacity per stage, not per company: arrivals after yield, mix-weighted touch hours, rework, productive hours and outside waits.
  • The queue multiplier, utilization over one minus utilization, is 4 at 80 percent busy, 9 at 90 and 19 at 95, before variability is added.
  • CRAN's published counts show submissions up 20.6 percent but hand-triggered actions up 46.5 percent, because first-time packages rose 62.8 percent.
  • In the illustrative model, a fifth more intake with a heavier case mix lifts rights review from 65 to 93 percent busy and its wait almost ninefold.
  • Waits on suppliers and buyers lengthen cycle time without using staff hours, so they belong in the cycle-time sum, not in utilization.
  • Fund capacity only at the stage whose planned utilization crosses a trigger band, and release the money on measured milestones.

What does dataset operations capacity planning have to answer?

Dataset operations capacity planning has to say which stage runs out of specialist hours first at the intake a plan expects, and how long work will wait in front of it. A headcount total cannot answer that. A model that treats each stage as a queue, with its own arrivals, touch time, variability and available hours, can.

The evidence it needs is ordinary operating data. For each stage, count the cases that arrive per week after the previous stage's yield, the hours a case takes by type, how often it comes back, the productive hours the people doing it really have, and the time it spends waiting on someone outside the team. A funding plan that shows those five numbers per stage, measured on completed cases, can be checked. A plan that shows only a hiring calendar cannot.

This page builds the model, tests its central warning on a public review pipeline whose specialist workload grew faster than its intake, and then answers the companion question of what a capacity model for a data licensing business must include. Every model input on the page is illustrative; public data carries its source and date.

What happened when CRAN's intake tilted toward first-time packages?

Its hand-review workload grew more than twice as fast as its intake. CRAN, the package repository for the R language, publishes period counts of submissions and review actions in the R Journal's Changes on CRAN columns by Kurt Hornik, Uwe Ligges and Achim Zeileis. From January to June 2025 it received 15,166 submissions; from January to June 2026, 18,293, a rise of 20.6 percent.

Over the same comparison, automated moves into the newbies folder, the queue where a person inspects a first-time package, rose 62.8 percent, from 3,241 to 5,275. Actions a CRAN team member triggered by hand rose 46.5 percent, from 5,800 to 8,497, and the share of final decisions taken by hand went from 36.0 to 43.7 percent. A plan that scaled reviewers to submissions would have funded a fifth more capacity for close to half again as much hand work.

The mix can also move the other way. From the first half of 2024 to the first half of 2025, submissions rose 4.0 percent while hand-triggered actions fell 18.7 percent. Either direction breaks a model that assumes a fixed ratio of specialist work to intake.

CRAN is a fair analog for dataset intake because the shape matches: automated checks clear routine cases, people inspect what the checks cannot settle, and a maintainer's reply is an outside wait the team does not control. Its repository policy describes a service run by volunteers, the CRAN team, and asks established packages to submit updates no more than about once every one to two months, an intake rule that protects reviewer time. The published counts by half-year:

What happened when CRAN's intake tilted toward first-time packages?
PeriodSubmissionsSent to first-time queueHand-triggered actionsDecisions taken by hand
Jan-Jun 202414,5843,5327,13142.8%
Jul-Dec 202412,8842,8625,50339.8%
Jan-Jun 202515,1663,2415,80036.0%
Jul-Dec 202514,9893,3005,88737.3%
Jan-Jun 202618,2935,2758,49743.7%

How long did first-time packages wait as that queue filled?

The median wait for a first look rose from 1.6 days in the first half of 2025 to 4.2 days in the first half of 2026, and to 8.6 days from July to September 2026. The source is r-hub's cransays project, which records the contents of CRAN's incoming folders about once an hour and commits each snapshot to a public history branch. This analysis reads 11,386 snapshots taken between December 1, 2024 and October 27, 2026.

A wait here is one continuous stay in the newbies folder, from the first snapshot that shows a package version there to the last one before it moves or disappears. Hourly snapshots miss stays shorter than an hour, so observed entries are 81 to 86 percent of the moves CRAN reports for the same three half-years, and every wait is a lower bound. Little's law, which says that under steady conditions average work in progress equals the arrival rate times the average time in the system, holds within about three packages for both first halves: 14.4 entries a day times a mean wait of 1.95 days predicts a queue of about 28, against 30.7 measured in 2025, and 24.9 a day times 4.1 days predicts about 102, against 98.8 in 2026.

Two patterns matter for planning. Arrivals were bursty: a Saturday or Sunday brought 35 to 60 percent of a Tuesday's average entries, and the variance of daily entries was 3.6 to 4.2 times their mean, where purely random (Poisson) arrivals give a ratio near 1. And backlogs did not drain. Across an 18-day break in the snapshot record in August 2025 the queue went from 42 to 79, and it averaged between 66.8 and 88.3 in each of the next four months, more than twice the first-half level, while CRAN's own counts of first-time moves and hand actions barely changed between the halves.

In the third quarter of 2026, entries held at about 25 a day, yet the monthly average queue climbed from 142 in June to 272 in October through the 27th. A queue that grows while arrivals hold flat means departures fell below arrivals. The snapshots cannot say why, and the R Journal column for that quarter was not in the journal's repository when checked on October 27, 2026. Measured waits by period:

How long did first-time packages wait as that queue filled?
PeriodEntries per dayAverage queueMedian wait, daysWaits over 7 days
Jan-Jun 202514.430.71.60.7%
Jul-Dec 202515.573.64.37.3%
Jan-Jun 202624.998.84.28.5%
Jul-Sep 202625.4209.48.666.7%

Why do queues jump before anyone looks overloaded?

Because the wait in front of a stage grows with its utilization divided by one minus its utilization, not in proportion to load. J. F. C. Kingman's 1961 note on the single server queue in heavy traffic is the origin of the standard approximation, which a May 2026 preprint by Silvia Bartolucci and Pierpaolo Vivo restates as the mean wait being roughly that utilization term, times the average of the squared coefficients of variation of arrival gaps and service times, times the mean service time.

The utilization term alone carries the warning. At 50 percent busy it equals 1; at 70 percent, 2.3; at 80 percent, 4; at 90 percent, 9; at 95 percent, 19. Moving a team from 80 to 90 percent busy more than doubles the wait, and moving it to 95 percent doubles it again, while the timesheets show only modestly fuller weeks. Bursty intake, like CRAN's, raises the variability term and scales every one of those numbers up.

Before relying on the approximation, this analysis checked it with a seeded simulation of a single queue with bursty arrivals (squared coefficient of variation 2.0) and variable service (1.44): 500,000 cases after a 50,000-case warm-up, five seeds. The formula understated the simulated mean wait by 9.1 percent at 65 percent utilization, 3.8 percent at 85 percent and 2.8 percent at 93 percent, close enough for planning and slightly optimistic. The practical rule that follows is an illustrative policy, not a law: plan specialist stages at or below 80 percent of productive hours, so a burst or an absence stretches the queue instead of multiplying it.

What should a capacity model for data licensing include?

It should include one row for every stage that consumes someone's time between qualified intake and accepted delivery, and for each row the arrivals after yield, touch hours by case type, rework share, productive hours, variability and the outside wait that delays a case without using capacity. Intake means qualified applications, not leads: a short self-serve assessment that screens out poor fits costs the applicant minutes and the specialists nothing.

The constraint differs by stage. Qualification is bound by analyst hours and by how complete the supplier's inventory is. Rights review is bound by a small pool of specialists and by case complexity: a company with several systems and many customer contracts usually needs more review than a single-system archive, which is why the model splits touch hours by case type. Agreements are bound by counterparties' signatures more than by staff. Preparation is bound by extraction access on the supplier's side and by processing and quality capacity, which may be shared with a partner. Acceptance review is bound by reviewer hours and by how often deliveries come back.

Outside waits need their own column. A supplier's signature, an onward buyer's commitment or a supplier's export window lengthens cycle time and raises work in progress through Little's law, but adding staff does not shorten it. The model's arithmetic, stage by stage:

  • Arrivals at a stage = qualified intake per week x the pass-on yields of every earlier stage, including outside steps such as buyer commitment.
  • Visits = arrivals x (1 + rework share), counting each return as another pass through the stage.
  • Mean touch hours = (1 - complex share) x simple-case hours + complex share x complex-case hours.
  • Utilization = visits x mean touch hours / productive specialist hours per week (calendar hours minus meetings, leave and non-case work).
  • Queue wait in weeks = utilization / (1 - utilization) x (arrival variability + service variability) / 2 x mean touch hours / productive hours per week.
  • Cycle time = the sum of queue waits, touch times and outside waits; work in progress = arrivals x cycle time.
What should a capacity model for data licensing include?
StageFill inWhat usually bindsEarly warning
Qualification and inventoryQualified applications per week; analyst hours per case by type; share sent back for missing systemsAnalyst hours; completeness of the supplier's inventoryFirst-look queue older than a week
Rights reviewCases reaching review; specialist hours for single-system and multi-system companies; rework shareA small specialist pool; complex-case shareComplex share rising faster than intake
Offer and agreementsCases reaching terms; staff hours per round; outside wait for each signatureCounterparty signatures more than staffSigned-but-idle deals piling up
Buyer commitment (outside step)Share of deals that win a commitment; wait from presentation to commitmentOnward buyers' review and budget cyclesWait lengthening while staff are idle
Preparation and quality checkDeliveries per week; engineer hours by archive size; partner capacity reserved; supplier access waitSupplier export access; processing and QA hoursReturn rate after quality check above plan
Acceptance reviewDeliveries reviewed per week; reviewer hours; share returned for fixesReviewer hours; delivery qualitySame dataset reviewed twice

Twenty percent more intake, heavier cases: the model filled in

With illustrative inputs, raising qualified intake by a fifth while the complex share climbs from 25 to 40 percent takes rights review from 65 to 93 percent busy and its queue from 1.8 to 15.8 working days, almost nine times as long. Complex cases take three times the rights hours of simple ones, so the stage's load rises 44 percent on 20 percent more cases, the same shape as CRAN's 20.6 percent more submissions and 46.5 percent more hand actions.

Nothing else in the example breaks, which is the point. Qualification reaches 84 percent and preparation 76 percent, both strained but stable. A plan that added an analyst because the qualification desk looked busy, or bought automation for preparation, would spend money at stages that are not the constraint while rights review sets the pace for every deal behind it.

Every number below is illustrative, chosen to show the mechanics and not drawn from any company. The arrivals are bursty (squared coefficient of variation 2.0), and each stage has one pooled specialist desk. The inputs, then the results:

  • Intake: 10 qualified applications a week with 25 percent complex cases; scenario 12 a week with 40 percent complex.
  • Qualification and inventory: 1.5 hours simple, 2.5 complex; 50 percent pass on; 10 percent rework; 30 productive hours a week; service variability 0.64.
  • Rights review: 3 hours simple, 9 complex; 70 percent pass on; 15 percent rework; 40 hours a week; variability 1.44.
  • Offer and agreements: 2 hours simple, 4 complex; 60 percent then win a buyer commitment; 10 percent rework; 20 hours a week; variability 0.64.
  • Preparation and quality check: 6 hours simple, 16 complex; 90 percent reach acceptance; 20 percent rework; 40 hours a week; variability 1.00.
  • Acceptance review: 3 hours simple, 6 complex; 10 percent rework; 15 hours a week; variability 0.49.
Twenty percent more intake, heavier cases: the model filled in
StageBusy share, base to scenarioQueue wait, working daysWhy it moved
Qualification and inventory64% to 84%0.7 to 2.1A fifth more cases, slightly longer complex screens
Rights review65% to 93%1.8 to 15.8Complex cases take 9 hours, not 3; load up 44%
Offer and agreements48% to 65%0.8 to 1.7More cases, mild mix effect
Preparation and quality check54% to 76%1.8 to 5.8Complex archives take 16 hours, not 6
Acceptance review52% to 70%1.7 to 4.0Follows preparation volume
A curve of queue wait multiplier against utilization, rising steeply past 80 percent, with rights review moving from 65 to 93 percent busy, beside a panel of five stages' busy shares before and after a fifth more intake.
Illustrative: the worked example's five stages before and after 20 percent more intake with a heavier case mix; the curve is utilization divided by one minus utilization.

Can more applications add cost without adding accepted supply?

Yes, whenever the added cases are the kind that need people. In CRAN's counts, publish decisions rose 29.2 percent from the first half of 2025 to the first half of 2026 while hand-triggered actions rose 46.5 percent, so hand actions per 100 publish decisions went from 76.8 to 87.0. More arrived and more was accepted, but each accepted package carried more specialist effort than before.

Hiring and automation can fail the same way. A hire at a stage that is not the bottleneck adds payroll and no throughput. Automation that speeds one step can lengthen the whole workflow: Bartolucci and Vivo's queueing model finds that faster AI-drafted tasks can raise system delay when errors escape review and return as rework, and that at low and moderate arrival rates their AI-routed system waited longer than the manual one.

The test that separates real capacity from added cost is a cohort table, traced from qualified intake to acceptance with fully loaded effort. Results for any specific company come from that company; the design is:

  • Group suppliers by the month they qualified, and keep each cohort's denominator fixed as cases move.
  • Record the date each case enters and leaves every stage, including outside waits, and leave open cases open (censored) rather than dropping them.
  • Log specialist hours by stage for every case, including rework and cases that later fail, so failed work stays in the cost.
  • Report accepted datasets per cohort and specialist hours per accepted dataset, by case type.
  • Compare cohorts before and after any hire or automation: hours per accepted dataset should fall, and the bottleneck's queue should shorten.

Where should the next operating dollar go?

To the stage whose planned utilization, at next period's intake and case mix, crosses the trigger band, and only once the inputs behind that forecast are measured rather than assumed. The bands below follow the utilization term above and are this article's illustrative policy; the milestones are evidence that reduces uncertainty about the next transaction, not revenue targets.

The table assumes the stage's touch hours, mix and outside waits come from completed cases. If they come from estimates, the first spend is measurement: time-stamp each stage and log hours for a full cohort before funding anything else.

Where should the next operating dollar go?
Planned utilization at the busiest stageReadingCapital decisionMilestone before money is released
Below 70%Spare capacity; queue multiplier under 2.3No capacity spend; fund qualified intakeTouch hours measured on at least 20 completed cases per type
70% to 85%Buffer thinning; multiplier 2.3 to 5.7Cross-train a second person; tighten pre-qualification; set a queue-age alarm90th-percentile queue age inside target for 8 straight weeks
85% to 100%Queue growing fast; small bursts cost weeksAdd capacity at that stage only, or cap intakeNew hours confirmed productive; queue wait falls within two cycle times
100% or more, or queue rising month on monthNo steady state; backlog grows without limitStop intake growth; triage by valueDepartures at or above arrivals for 4 straight weeks

How SourceX's published deal steps map onto the model

SourceX's how-it-works page lists nine steps from application to payment; all but the last map to a row of the model or to an outside wait, since the model stops at acceptance. Its first step is a short company assessment that needs no data uploads, a screen that costs specialists nothing. Within qualification, the page says SourceX checks fit, completes the inventory with the company, runs the rights check and sets the processing route: the qualification and rights rows, both specialist time.

The indicative offer and the deal alignment are mostly staff work; the Master agreement and the per-deal Dataset Deal Confirmation are signatures, which makes them outside waits on the supplier. In the commercial commitment step SourceX seeks commitment on the approved terms and signs its own onward agreement before instructing processing, an outside wait on onward counterparties with its own pass-on yield. Preparation runs through approved processing routes, with de-identification done by SourceX directly or through its processing partner, so part of that row's capacity sits with a partner. SourceX then reviews the delivered data before Closing, the acceptance row.

What the page does not show is volume: arrivals, hours, utilization and staffing are not public, and SourceX reports company and commercial metrics to investors directly. Because the company raises money under Rule 506(c), any investment is limited to verified accredited investors; the materials request is at sourcex.si/investors. Questions that test a plan against this model:

Checklist · 5 items

  1. Which step had the highest utilization last quarter, and at what qualified intake would it pass 85 percent?
  2. How many completed cases, by type, are the touch-hour estimates based on?
  3. How is processing partner capacity reserved, and what happens to queue time when the partner is full?
  4. What is the distribution of outside waits per step, and how much work in progress do they hold?
  5. How have specialist hours per accepted dataset moved across qualification cohorts?

0 of 5 ticked. Tick each item as you go.

What could make this capacity model wrong?

The model is only as good as its inputs and its assumption of steady conditions. These are the ways it most often misleads:

At a glance

What could make this capacity model wrong

  • Mix drift: CRAN's hand workload rose 46.5 percent on 20.6 percent more submissions in one year and fell 18.7 percent on 4.0 percent more the year before, so touch hours by type need re-estimating every period.
  • Understated variability: the Kingman formula ran about 3 to 9 percent below simulation even with tidy distributions, and rare very large archives make real service times heavier-tailed than any gamma curve.
  • Phantom capacity: productive hours are well below calendar hours, and absences or breaks remove capacity in blocks; CRAN's queue never returned to its early-2025 level after August 2025.
  • Outside waits mistaken for staff shortages: if most cycle time sits with suppliers or buyers, hiring shortens nothing and adds cost.
  • Automation that moves the bottleneck: Bartolucci and Vivo's model shows faster drafting can lengthen queues through rework that escapes review.
  • Quality slipping under load: the same model finds reviewers become more selective about what they check when congested, the riskiest moment to reduce scrutiny.
  • Analog limits: CRAN is a volunteer-run, open-source review service with small, fairly uniform cases; a commercial licensing pipeline has larger, more varied cases and paid counterparties.

Methods note: data, counting rules and what was not measured

CRAN workload: the R Journal Changes on CRAN columns read from the journal's site repository (rjournal/rjournal.github.io, commit 0d29f348, checked October 27, 2026), issues 2024-1 through 2026-2, covering January 1, 2024 to June 30, 2026. Submissions and manual actions are the columns' stated totals; first-time moves are the auto-processed actions into the newbies folder; decisions taken by hand are manual archive plus manual publish decisions over all four decision cells. Half-years sum the quarters; growth rates and shares are computed on integers and rounded half up once.

Queue waits: r-hub/cransays history branch, commit 6ccb491b, 11,386 files named cran-incoming-v5 with timestamps from December 1, 2024 to October 27, 2026, read as UTC from the file name. Item key is package and version; 2,610 duplicate rows where a key appeared twice in one snapshot were dropped, keeping the first in file order. A stay continues across snapshots missing the key unless the key is absent for more than 24 hours, and ends when the key appears in another folder. Waits run from first to last snapshot in the stay, so they are lower bounds; medians are exact (mean of the two middle values when the count is even), 90th percentiles are nearest-rank, and days are rounded half up to one decimal. About a quarter of stays are re-entries of a version already seen; medians for first stays alone differ by at most 0.1 days, and dropping stays that span a snapshot gap of more than 12 hours moves no median by as much as half a day. Daily dispersion uses days with at least 12 snapshots. A second break in the record, from August 5 to 19, 2026, cuts 256 third-quarter stays short (the queue read 259 before it and 14 after). The saved data keeps only stay start and end times, with no package names or team initials.

Model and simulation: all inputs are illustrative. The simulation is a single-queue Lindley recursion, W(n+1) = max(0, W(n) + S(n) - A(n)), where S(n) is case n's service time and A(n) the gap before the next arrival, both gamma-distributed, Python 3.13 random.Random with seeds 1 to 5, interarrival time drawn before service time for each case. Not measured: CRAN staffing or hours, the reasons behind any queue change, and any SourceX volume, hours, utilization or timing, none of which is public.

Frequently asked questions

What utilization should a data operations team plan for?

Plan specialist stages at or below about 80 percent of productive hours; that ceiling is a planning policy, not a law. At 80 percent the utilization term equals 4, at 90 percent it is 9 and at 95 percent 19, before bursty arrivals or variable cases scale it further. Stages that only wait on outside parties do not need a utilization target at all.

How is a capacity model different from a headcount plan?

A headcount plan counts people; a capacity model counts arrivals, hours and waits per stage and finds the one stage that limits throughput. Two plans with the same headcount can have very different queues if one puts its people at the bottleneck. The model also separates outside waits, which staff cannot shorten, from staff-bound queues.

Why did CRAN's review queue grow faster than its submissions?

Because the growth came mostly from the case type that needs people. From early 2025 to early 2026, CRAN's submissions rose 20.6 percent but first-time packages sent to manual inspection rose 62.8 percent, and hand-triggered actions 46.5 percent. In the cransays snapshots the average first-time queue went from 30.7 to 98.8.

Does automation remove the bottleneck in data operations?

It removes work at the step it automates and can move the constraint elsewhere. If automated output needs review and its errors come back as rework, total delay can rise, which is what Bartolucci and Vivo's 2026 queueing model finds at low and moderate arrival rates. Check the bottleneck's queue before and after, not the automated step's speed.

Does SourceX publish its throughput or utilization?

No. SourceX's deal steps are public on its how-it-works page, but its volumes, hours, staffing and utilization are not. It shares company and commercial metrics with investors directly through the materials request at sourcex.si/investors.

Sources

  • The R Journal's Changes on CRAN column for January to June 2026 reports 18,293 package submissions and 30,170 actions, of which 21,673 were auto processed and 8,497 manual. Source
  • The Changes on CRAN columns for issues 2024-1 through 2026-2 report each period's submissions, auto and manual actions by destination folder (including newbies) and auto and manual archive and publish decisions. Source
  • The cransays dashboard is updated hourly from CRAN's incoming folder; newbies is the queue for manual inspection of first-time submissions and waiting means the CRAN team awaits the maintainer's answer. Source
  • The cransays history branch holds the hourly snapshots of CRAN's incoming folders analyzed here (package, version, snapshot time and folder for every queued submission). Source
  • The CRAN Repository Policy says CRAN is maintained by volunteers, the CRAN team, and asks established packages to limit update submissions to about one every one to two months. Source
  • John D. C. Little (1961) proved that, with finite means and stationary processes, the average number in a queueing system equals the arrival rate times the average time in the system (L = lambda W). Source
  • J. F. C. Kingman's 1961 research note The single server queue in heavy traffic, the origin of the heavy-traffic waiting time approximation. Source
  • Bartolucci and Vivo (2026) restate Kingman's approximation and model how AI-drafted tasks can lengthen queues through rework that escapes review, with longer waits than manual work at low and moderate arrival rates. Source
  • SourceX's published deal steps: it checks fit, completes the inventory, runs the rights check and sets the processing route; seeks commitment and signs its onward agreement before processing; reviews delivered data before Closing. Source
  • SourceX shares company and commercial metrics with investors directly, through an investor materials request. Source

Request SourceX investor materials

SourceX shares company and commercial metrics with investors directly. Any offering is made only to verified accredited investors.

See if your company qualifies