API rate limits and full-history exports: how long a large pull takes

The basic formula for export time#

The basic formula for API export time is the number of calls divided by the calls allowed per minute. The number of calls is the record count divided by the page size, plus any extra calls for child records, attachments and retries.

The inputs come from two places. Record counts come from the system itself, usually through reports or admin screens. Page sizes and rate limits come from the vendor's developer documentation and your plan, and they can differ by endpoint, app type and plan tier.

• 1List calls = records divided by records returned per page.
• 2Child calls = one or more calls per parent record, unless the API returns children in bulk.
• 3Minutes of calling = total calls divided by calls allowed per minute.
• 4Elapsed time = minutes of calling plus waits for limit resets, retries and any windows when the pull must pause.

Which inputs change the estimate most?#

The inputs that change an export estimate most are record counts per object, page sizes, the binding rate limit and attachment sizes. Get the counts first: ask the system owner for records per object and per year before any engineer writes code.

When a vendor sets both a short-window limit and a daily cap, work out the time under each and use the longer result. A daily cap means the pull cannot finish faster than total calls divided by the daily allowance, however quickly each minute runs. That longer figure is the one to give finance and leadership when they ask when the export will be done.

Why child records multiply the call count#

Child records multiply the call count because many APIs return a parent list cheaply but need a separate request for each parent's details. A help desk may list tickets in large pages but return comments one ticket at a time. A code platform may list pull requests in pages but need further calls for each one's reviews, review comments and timeline.

This pattern turns a pull that looks short on paper into a long one. When a system holds many more child records than parents, per-parent calls dominate the estimate and the page size of the parent list barely matters.

Look for endpoints that return children in bulk, such as incremental exports of all comments by date, or options that include related records in the parent response. Finding one is usually worth more than tuning the code.

Bulk API versus REST: which to use#

A bulk or asynchronous export API is usually the better choice for a full-history pull when the vendor offers one, and paginated REST is the fallback for objects bulk endpoints do not cover. Many large platforms offer both, with separate limits.

Most real pulls mix methods: bulk jobs for large flat objects, REST for child details and attachments, and an incremental pass at the end to catch records that changed while the job ran.

What do published limits look like in practice?#

Published export limits come in several shapes: calls per hour, exports per minute, items per export and exports per week. Each shape constrains the plan differently, which is why one formula needs different inputs per system.

The examples below are taken from vendor documentation and change over time. Use them to recognize the type of limit you face, then confirm the current figure for your plan before you build the schedule.

Illustrative: estimating a help desk pull#

Illustrative: a fictional vertical software company wants every ticket and comment from its help desk before a platform change. The figures below are invented to show the method; they are not any vendor's limits.

The archive holds 400,000 tickets. The list endpoint returns 100 tickets per page, so listing takes 4,000 calls. Comments come back one ticket at a time, which adds 400,000 calls. At an allowance of 200 calls per minute, listing takes about 20 minutes and comments take about 2,000 minutes, more than 33 hours of uninterrupted calling before retries or attachments.

The engineer then finds an incremental endpoint that returns comments for all tickets by date. With about four comments per ticket and 1,000 comments per page, the comment pull drops to about 1,600 calls. The figures are made up, but the shape is typical: per-parent calls set the timeline, and the right endpoint matters more than faster code.

What else stretches a large pull#

Several factors beyond the published rate limit stretch a large pull, and most appear only once the job is running. Build for them from the start.

Checkpoint progress so a failed job resumes from the last completed window rather than starting over. Store raw API responses as well as transformed files, so a later mapping error does not force a second pull.

• 1Limit responses: back off and retry when the API signals a limit, and log every retry.
• 2Shared limits: production integrations may draw on the same allowance, so schedule heavy phases off-hours.
• 3Deep pagination: some APIs cap how far offset paging goes, so split the pull into date windows or use cursors.
• 4Token expiry: long jobs need credentials that refresh without manual steps.
• 5Attachments: file downloads can carry separate limits and much larger payloads.
• 6Changes during the pull: records edited mid-pull need a final incremental pass.

How SourceX scopes a large export#

SourceX scopes a large export during Supply, the first step of the SourceX five-step transaction, using metadata the company already has: systems, object counts, years of history and known export routes. No data moves during that assessment.

If a package goes ahead, the export runs in the company's environment and the data stays in its own storage or ships on encrypted drives. The SourceX Evidence Packet records provenance, including the export method and date, so a buyer can see how the dataset was produced.