Skip to content

AI uses for records

API integration failure dataset: evidence when the real API breaks

By SourceX Editorial · Updated

9 sections

Short answer

Include the contract version the code relied on, the mock or fixture that let local tests pass, the real failing request and response with secrets removed, the environment and dependency versions, and the fix with tests that fail before and pass after against a replay of that recorded response. Without the observed exchange, the task cannot show why the passing tests were wrong or confirm the fix.

System export

API integration failure

This page covers API integration failure dataset: evidence when the real API breaks.Illustrative

Key takeaways

  • A passing local test only proves the code handles the response the developer expected.
  • Store the contract version and request headers the code used, because providers change defaults over time.
  • Grade against a replay of the recorded production exchange, never the live API.
  • Fail-to-pass and pass-to-pass tests, in a pinned environment, make the fix verifiable by someone else.
  • Drop records whose failing field would not survive redaction, and check code and API-terms rights separately.

What evidence should an integration task include when local tests pass but the real API fails?

It should include five linked things: the contract the code relied on, at the version in force; the mock or fixture that let local tests pass; the real request and response that failed, with secrets removed; the environment and dependency versions; and the fix with tests that fail before it and pass after it against the recorded real behavior. Remove any one and the task stops proving why the passing tests were wrong.

The reason is structural. A local test suite for an integration encodes an assumption about the provider. Pact's documentation states this directly for consumer tests: they check that the consumer handles the expected response, assuming the provider returns it. When production disagrees, the valuable record is the gap between that assumption and what the provider actually sent.

An API integration failure dataset built this way teaches something a patch alone cannot: how to read a contract skeptically, how to tell a provider behavior from a client bug, and how to write a regression test that would have caught it.

Why do integrations pass against a mock and break against the provider?

Because a mock returns what the developer expected, and the real provider returns what it actually does under load, at a different version, with real permissions and real data. Most failures fall into a small number of classes, and each class needs its own evidence to be gradeable.

Classifying the failure first also decides who owns the fix. A rate limit or retry problem is almost always the client's to solve, a silent version change may be the provider's to document, and an undeclared status code is shared: the provider did not list it and the client did not plan for it. A task that records the class lets a grader reward the right diagnosis, not only a green test run.

The table maps the common classes to the proof a task must carry. The examples are patterns, not claims about any specific provider beyond the cited documentation.

Why do integrations pass against a mock and break against the provider?
Failure classWhat the mock assumedWhat the provider didEvidence the task needs
Undeclared status codeOnly the documented success and error codesReturned a code the description never listedCaptured status line and the contract's responses section
Rate limitingUnlimited callsAnswered 403 or 429 with reset or retry headersResponse headers and request timing across the burst
Version driftThe version in the fixtureServed a different default versionRequest headers sent, provider version policy on that date
Error body shapeA flat message stringReturned a structured problem objectRaw error body and the media type header
Unsafe retryEach call either succeeds or fails cleanlyAccepted the first call, then timed outBoth requests, idempotency headers, provider object list
Permission scopeA test token with full accessRejected or filtered for the real credential's scopeScope list of the credential type, redacted
Pagination and orderingOne short page in a fixed orderMany pages with cursor rulesPage sequence and cursor values, truncated

Which version of the API contract was the code written against?

The task has to say, because contracts move and the failure is often that the code and the contract drifted apart. Record the OpenAPI description, SDK version or documentation snapshot the code was written against, plus the date, so a reviewer can check whether the provider broke its promise or the client never read it.

The OpenAPI Specification is a useful anchor here and also a warning. Version 3.2.0 says a description is not necessarily expected to cover every possible HTTP response code, because they may not be known in advance, and offers a default response for undeclared codes. A client generated from an OpenAPI file can therefore pass every schema-based test and still meet a code it never handled.

Version headers matter as much as the description. GitHub's REST documentation says clients should send the X-GitHub-Api-Version header, that requests without it fall back to a default version, and that versions are supported for 24 months after a newer one is released. An integration that omits the header can change behavior without a single line of its own code changing, so the task should store the headers actually sent.

What makes an observed failure reproducible without calling the live API?

A captured exchange, replayed. Grading cannot depend on calling a third-party API that may have changed, rate-limited the grader or been retired, so the task stores the real request and response, with secrets replaced, and the test harness serves that recorded response in place of the provider.

Capture headers, not only bodies. GitHub documents that exceeding its rate limits returns 403 or 429, that the x-ratelimit-remaining header shows calls left and that a retry-after header, when present, says how many seconds to wait. It also warns that remaining counts can vary between responses because requests are handled in several regions, so the task should test backoff logic, not an exact counter. RFC 6585 defines 429 Too Many Requests and allows a Retry-After header.

Capture the error body exactly. RFC 9457 defines a standard problem details format with members such as type, title, status, detail and instance, served as application/problem+json. A client that parses only a message string fails silently on that shape, and the only proof is the raw body.

Keep timing and correlation. Timestamps, request IDs the provider returned and the client's retry log are what separate a provider outage from a client that retried too fast.

Which fields make up the API contract and observed failure task?

This is the asset. One task equals one observed failure with everything a reviewer needs to reproduce it offline and verify a fix. Use the fields as the template; the right column says how each field is checked so the task can be graded without a human reading logs.

At a glance

Which fields make up the API contract and observed failure task

  • Balance failure classes so rate limiting, which is common and easy to capture, does not crowd out version drift and unsafe retries.
  • Include several providers, so a model cannot pass by memorizing one vendor's error format.
  • Keep some tasks where the correct diagnosis is a client bug despite a provider error, and some where the provider changed behavior despite a clean client.
  • Add a small set of no-fix tasks, where the right answer is that the failure was a transient outage and no code change is justified.
  • Hold out whole repositories for evaluation, not individual tasks, because shared helper code leaks fixes between tasks from the same codebase.
Which fields make up the API contract and observed failure task?
FieldContentsUsual sourceHow it is verified
RequirementTicket or issue describing the intended behavior, with versionJira, Linear or GitHub issueReviewer confirms the fix targets this requirement
Contract snapshotOpenAPI file, SDK version or docs capture with dateRepository, vendored spec, docs archiveHash stored; diff against later versions recorded
Base commitCode before the fix, with lockfileGit historyBuilds in the pinned environment
Passing local evidenceMock, fixture or contract test that passedTest suite and CI logRuns green on the base commit
Observed failureRedacted request, response, headers, timestampsGateway logs, APM traces, client logsReplayed by the harness in place of the provider
DiagnosisFailure class and the contract clause or behavior involvedIncident note or pull request discussionMatches the replayed exchange
Fix and reviewPatch, review comments and approvalPull request in GitHub or GitLabLinked to the base commit
Fail-to-pass testsNew tests using the recorded responseTest patchFail on base commit, pass with fix
Pass-to-pass testsExisting tests that must stay greenTest suitePass before and after
Production outcomeError rate or reconciliation after deployMonitoring or provider dashboard exportRecorded as evidence, not used for grading

Can a separate reviewer rebuild the environment and confirm the fix?

Only if the task ships a pinned environment and a test boundary that never touches the network. The SWE-bench dataset format is the clearest public model: each instance records a base commit, the gold patch, a test patch and two lists of tests, FAIL_TO_PASS and PASS_TO_PASS, and the project uses Docker for reproducible evaluation.

Integration tasks add one requirement those code tasks rarely need: the provider must be replaced by a replay of the recorded exchange. That keeps grading stable, keeps credentials out of the environment and avoids sending traffic to someone else's service.

Contract testing tools show what can and cannot be proven offline. Pact's provider verification replays each request against the provider and checks the response contains at least what the consumer expected, which works when you own the provider. For a third-party API you cannot run that step, so the recorded production exchange is the closest available stand-in for the provider's side of the contract, and its capture date must travel with it.

How did a retry create two orders? An illustrative case

This case is illustrative, not drawn from a real company. A commerce team's client creates orders with a payment provider. Local tests mock the create call as an immediate success. In production, one create call takes longer than the client's 10-second timeout, the client retries, and the provider ends up with two orders for one checkout.

Linked evidence: the client log with both requests 12 seconds apart and no idempotency header; the provider's object list showing two records; the contract snapshot, which documents idempotency keys; and the fixture, which never models a timeout. Stripe's documentation, for example, describes idempotency keys that let a client safely retry a create request: once execution begins, the first result is saved and replayed for retries with the same key, and a 500 response should be treated as indeterminate until reconciled.

Decision and fix: generate one idempotency key per checkout, persist it, and send it on every attempt. The fail-to-pass test replays the recorded timeout followed by success and asserts a single order. The pass-to-pass tests confirm normal checkouts still work.

Counterexample: the same client retries after a 429 that arrived before the provider began executing the request. Stripe states that requests rejected before execution, such as by rate limiting, do not cache a result, so a retry after waiting is safe. A model that learns to block every retry has learned the wrong rule.

Which failure records cannot become tasks?

Records missing the observed exchange, records whose payloads cannot be cleaned, and failures that were never really contract mismatches. Without the real response, the task reduces to a patch and a story, and nothing shows the passing tests were wrong.

Provider outages with no contract change belong in a different set; they teach resilience, not contract reading. Flaky failures that never reproduced should be excluded or labeled as such. Payloads holding customer personal data, card data or access tokens need to be redacted to typed placeholders, and if redaction would remove the field that caused the failure, the record is not usable.

Check two kinds of rights before anything leaves the company. The code may include customer-owned or contractor-written components, and the API provider's terms may limit storing or sharing its responses. Neither question is answered by removing personal data.

Preparing integration failure history for a licensing review

Software companies with years of integration incidents, pull requests and gateway logs hold the linked evidence this kind of task needs. The practical sequence is to confirm rights in the code and the response data, pick a sample of incidents where the real exchange was captured, and agree redaction rules with security and the engineering lead before export.

Software companies that go ahead can do so with SourceX, which covers rights review, contracting, preparation and delivery when operational records are licensed to AI developers. Large repositories and log archives can stay in the company's own storage or ship on encrypted drives rather than being uploaded, and the first step is a short assessment that needs no data upload.

Frequently asked questions

Why do API integrations pass tests locally but fail in production?

Local tests usually run against mocks or fixtures that return what the developer expected. The real provider can return undeclared status codes, rate limits, a different version or a different error shape. The test passed because the assumption was never checked against the provider.

Is a contract test enough to prevent these failures?

It helps when you control both sides, because provider verification replays the consumer's expectations against the real provider. For third-party APIs you usually cannot run that step, so recorded production exchanges and version pinning carry more of the weight.

Should a benchmark call the live API during grading?

No. Live calls make results depend on the provider's current state, can hit rate limits and require real credentials. Replay the recorded request and response instead, and store the capture date so reviewers know which provider behavior the task reflects.

What should be redacted from captured API traffic?

Access tokens, API keys, cookies, signatures, personal data and anything that identifies a customer account. Replace them with typed placeholders so the structure stays intact. If the failing field cannot be kept after redaction, drop the record.

How is an API integration failure dataset different from tool-call error data?

Tool-call error data focuses on how an agent recovers within a session. An integration failure task asks why engineered code broke against a real API and requires a fix verified by tests, so it carries a contract snapshot, a commit, a patch and a replayable exchange.

Sources

  • The OpenAPI Specification 3.2.0 says a description need not cover every HTTP response code and provides a default response for undeclared codes. Source
  • Pact consumer tests assume the provider returns the expected response; provider verification replays each request against the provider and compares the actual response. Source
  • GitHub's REST API returns 403 or 429 when rate limits are exceeded, exposes x-ratelimit headers and retry-after, and notes remaining counts can vary between responses. Source
  • GitHub's REST API uses the X-GitHub-Api-Version header, defaults unversioned requests and supports each version for 24 months after a newer one is released. Source
  • SWE-bench instances record a base commit, gold patch, test patch and FAIL_TO_PASS and PASS_TO_PASS test lists. Source
  • SWE-bench uses Docker for reproducible evaluations. Source
  • RFC 9457 defines problem details for HTTP APIs with type, title, status, detail and instance members and the application/problem+json media type. Source
  • RFC 6585 defines the 429 Too Many Requests status code, which may include a Retry-After header. Source
  • Stripe idempotency keys let clients retry safely; results are saved once execution begins, 500s are indeterminate, and requests rejected before execution do not cache a result. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies