AI uses for records
API integration failure dataset: evidence when the real API breaks
By SourceX Editorial · Updated
9 sections
Short answer
Include the contract version the code relied on, the mock or fixture that let local tests pass, the real failing request and response with secrets removed, the environment and dependency versions, and the fix with tests that fail before and pass after against a replay of that recorded response. Without the observed exchange, the task cannot show why the passing tests were wrong or confirm the fix.
System export
API integration failure
Key takeaways
- A passing local test only proves the code handles the response the developer expected.
- Store the contract version and request headers the code used, because providers change defaults over time.
- Grade against a replay of the recorded production exchange, never the live API.
- Fail-to-pass and pass-to-pass tests, in a pinned environment, make the fix verifiable by someone else.
- Drop records whose failing field would not survive redaction, and check code and API-terms rights separately.
What evidence should an integration task include when local tests pass but the real API fails?
It should include five linked things: the contract the code relied on, at the version in force; the mock or fixture that let local tests pass; the real request and response that failed, with secrets removed; the environment and dependency versions; and the fix with tests that fail before it and pass after it against the recorded real behavior. Remove any one and the task stops proving why the passing tests were wrong.
The reason is structural. A local test suite for an integration encodes an assumption about the provider. Pact's documentation states this directly for consumer tests: they check that the consumer handles the expected response, assuming the provider returns it. When production disagrees, the valuable record is the gap between that assumption and what the provider actually sent.
An API integration failure dataset built this way teaches something a patch alone cannot: how to read a contract skeptically, how to tell a provider behavior from a client bug, and how to write a regression test that would have caught it.
Why do integrations pass against a mock and break against the provider?
Because a mock returns what the developer expected, and the real provider returns what it actually does under load, at a different version, with real permissions and real data. Most failures fall into a small number of classes, and each class needs its own evidence to be gradeable.
Classifying the failure first also decides who owns the fix. A rate limit or retry problem is almost always the client's to solve, a silent version change may be the provider's to document, and an undeclared status code is shared: the provider did not list it and the client did not plan for it. A task that records the class lets a grader reward the right diagnosis, not only a green test run.
The table maps the common classes to the proof a task must carry. The examples are patterns, not claims about any specific provider beyond the cited documentation.
| Failure class | What the mock assumed | What the provider did | Evidence the task needs |
|---|---|---|---|
| Undeclared status code | Only the documented success and error codes | Returned a code the description never listed | Captured status line and the contract's responses section |
| Rate limiting | Unlimited calls | Answered 403 or 429 with reset or retry headers | Response headers and request timing across the burst |
| Version drift | The version in the fixture | Served a different default version | Request headers sent, provider version policy on that date |
| Error body shape | A flat message string | Returned a structured problem object | Raw error body and the media type header |
| Unsafe retry | Each call either succeeds or fails cleanly | Accepted the first call, then timed out | Both requests, idempotency headers, provider object list |
| Permission scope | A test token with full access | Rejected or filtered for the real credential's scope | Scope list of the credential type, redacted |
| Pagination and ordering | One short page in a fixed order | Many pages with cursor rules | Page sequence and cursor values, truncated |
Which version of the API contract was the code written against?
The task has to say, because contracts move and the failure is often that the code and the contract drifted apart. Record the OpenAPI description, SDK version or documentation snapshot the code was written against, plus the date, so a reviewer can check whether the provider broke its promise or the client never read it.
The OpenAPI Specification is a useful anchor here and also a warning. Version 3.2.0 says a description is not necessarily expected to cover every possible HTTP response code, because they may not be known in advance, and offers a default response for undeclared codes. A client generated from an OpenAPI file can therefore pass every schema-based test and still meet a code it never handled.
Version headers matter as much as the description. GitHub's REST documentation says clients should send the X-GitHub-Api-Version header, that requests without it fall back to a default version, and that versions are supported for 24 months after a newer one is released. An integration that omits the header can change behavior without a single line of its own code changing, so the task should store the headers actually sent.
What makes an observed failure reproducible without calling the live API?
A captured exchange, replayed. Grading cannot depend on calling a third-party API that may have changed, rate-limited the grader or been retired, so the task stores the real request and response, with secrets replaced, and the test harness serves that recorded response in place of the provider.
Capture headers, not only bodies. GitHub documents that exceeding its rate limits returns 403 or 429, that the x-ratelimit-remaining header shows calls left and that a retry-after header, when present, says how many seconds to wait. It also warns that remaining counts can vary between responses because requests are handled in several regions, so the task should test backoff logic, not an exact counter. RFC 6585 defines 429 Too Many Requests and allows a Retry-After header.
Capture the error body exactly. RFC 9457 defines a standard problem details format with members such as type, title, status, detail and instance, served as application/problem+json. A client that parses only a message string fails silently on that shape, and the only proof is the raw body.
Keep timing and correlation. Timestamps, request IDs the provider returned and the client's retry log are what separate a provider outage from a client that retried too fast.
Which fields make up the API contract and observed failure task?
This is the asset. One task equals one observed failure with everything a reviewer needs to reproduce it offline and verify a fix. Use the fields as the template; the right column says how each field is checked so the task can be graded without a human reading logs.
At a glance
Which fields make up the API contract and observed failure task
- Balance failure classes so rate limiting, which is common and easy to capture, does not crowd out version drift and unsafe retries.
- Include several providers, so a model cannot pass by memorizing one vendor's error format.
- Keep some tasks where the correct diagnosis is a client bug despite a provider error, and some where the provider changed behavior despite a clean client.
- Add a small set of no-fix tasks, where the right answer is that the failure was a transient outage and no code change is justified.
- Hold out whole repositories for evaluation, not individual tasks, because shared helper code leaks fixes between tasks from the same codebase.
| Field | Contents | Usual source | How it is verified |
|---|---|---|---|
| Requirement | Ticket or issue describing the intended behavior, with version | Jira, Linear or GitHub issue | Reviewer confirms the fix targets this requirement |
| Contract snapshot | OpenAPI file, SDK version or docs capture with date | Repository, vendored spec, docs archive | Hash stored; diff against later versions recorded |
| Base commit | Code before the fix, with lockfile | Git history | Builds in the pinned environment |
| Passing local evidence | Mock, fixture or contract test that passed | Test suite and CI log | Runs green on the base commit |
| Observed failure | Redacted request, response, headers, timestamps | Gateway logs, APM traces, client logs | Replayed by the harness in place of the provider |
| Diagnosis | Failure class and the contract clause or behavior involved | Incident note or pull request discussion | Matches the replayed exchange |
| Fix and review | Patch, review comments and approval | Pull request in GitHub or GitLab | Linked to the base commit |
| Fail-to-pass tests | New tests using the recorded response | Test patch | Fail on base commit, pass with fix |
| Pass-to-pass tests | Existing tests that must stay green | Test suite | Pass before and after |
| Production outcome | Error rate or reconciliation after deploy | Monitoring or provider dashboard export | Recorded as evidence, not used for grading |
Can a separate reviewer rebuild the environment and confirm the fix?
Only if the task ships a pinned environment and a test boundary that never touches the network. The SWE-bench dataset format is the clearest public model: each instance records a base commit, the gold patch, a test patch and two lists of tests, FAIL_TO_PASS and PASS_TO_PASS, and the project uses Docker for reproducible evaluation.
Integration tasks add one requirement those code tasks rarely need: the provider must be replaced by a replay of the recorded exchange. That keeps grading stable, keeps credentials out of the environment and avoids sending traffic to someone else's service.
Contract testing tools show what can and cannot be proven offline. Pact's provider verification replays each request against the provider and checks the response contains at least what the consumer expected, which works when you own the provider. For a third-party API you cannot run that step, so the recorded production exchange is the closest available stand-in for the provider's side of the contract, and its capture date must travel with it.
How did a retry create two orders? An illustrative case
This case is illustrative, not drawn from a real company. A commerce team's client creates orders with a payment provider. Local tests mock the create call as an immediate success. In production, one create call takes longer than the client's 10-second timeout, the client retries, and the provider ends up with two orders for one checkout.
Linked evidence: the client log with both requests 12 seconds apart and no idempotency header; the provider's object list showing two records; the contract snapshot, which documents idempotency keys; and the fixture, which never models a timeout. Stripe's documentation, for example, describes idempotency keys that let a client safely retry a create request: once execution begins, the first result is saved and replayed for retries with the same key, and a 500 response should be treated as indeterminate until reconciled.
Decision and fix: generate one idempotency key per checkout, persist it, and send it on every attempt. The fail-to-pass test replays the recorded timeout followed by success and asserts a single order. The pass-to-pass tests confirm normal checkouts still work.
Counterexample: the same client retries after a 429 that arrived before the provider began executing the request. Stripe states that requests rejected before execution, such as by rate limiting, do not cache a result, so a retry after waiting is safe. A model that learns to block every retry has learned the wrong rule.
Which failure records cannot become tasks?
Records missing the observed exchange, records whose payloads cannot be cleaned, and failures that were never really contract mismatches. Without the real response, the task reduces to a patch and a story, and nothing shows the passing tests were wrong.
Provider outages with no contract change belong in a different set; they teach resilience, not contract reading. Flaky failures that never reproduced should be excluded or labeled as such. Payloads holding customer personal data, card data or access tokens need to be redacted to typed placeholders, and if redaction would remove the field that caused the failure, the record is not usable.
Check two kinds of rights before anything leaves the company. The code may include customer-owned or contractor-written components, and the API provider's terms may limit storing or sharing its responses. Neither question is answered by removing personal data.
Preparing integration failure history for a licensing review
Software companies with years of integration incidents, pull requests and gateway logs hold the linked evidence this kind of task needs. The practical sequence is to confirm rights in the code and the response data, pick a sample of incidents where the real exchange was captured, and agree redaction rules with security and the engineering lead before export.
Software companies that go ahead can do so with SourceX, which covers rights review, contracting, preparation and delivery when operational records are licensed to AI developers. Large repositories and log archives can stay in the company's own storage or ship on encrypted drives rather than being uploaded, and the first step is a short assessment that needs no data upload.
Frequently asked questions
Why do API integrations pass tests locally but fail in production?
Local tests usually run against mocks or fixtures that return what the developer expected. The real provider can return undeclared status codes, rate limits, a different version or a different error shape. The test passed because the assumption was never checked against the provider.
Is a contract test enough to prevent these failures?
It helps when you control both sides, because provider verification replays the consumer's expectations against the real provider. For third-party APIs you usually cannot run that step, so recorded production exchanges and version pinning carry more of the weight.
Should a benchmark call the live API during grading?
No. Live calls make results depend on the provider's current state, can hit rate limits and require real credentials. Replay the recorded request and response instead, and store the capture date so reviewers know which provider behavior the task reflects.
What should be redacted from captured API traffic?
Access tokens, API keys, cookies, signatures, personal data and anything that identifies a customer account. Replace them with typed placeholders so the structure stays intact. If the failing field cannot be kept after redaction, drop the record.
How is an API integration failure dataset different from tool-call error data?
Tool-call error data focuses on how an agent recovers within a session. An integration failure task asks why engineered code broke against a real API and requires a fix verified by tests, so it carries a contract snapshot, a commit, a patch and a replayable exchange.
Sources
- The OpenAPI Specification 3.2.0 says a description need not cover every HTTP response code and provides a default response for undeclared codes. Source
- Pact consumer tests assume the provider returns the expected response; provider verification replays each request against the provider and compares the actual response. Source
- GitHub's REST API returns 403 or 429 when rate limits are exceeded, exposes x-ratelimit headers and retry-after, and notes remaining counts can vary between responses. Source
- GitHub's REST API uses the X-GitHub-Api-Version header, defaults unversioned requests and supports each version for 24 months after a newer one is released. Source
- SWE-bench instances record a base commit, gold patch, test patch and FAIL_TO_PASS and PASS_TO_PASS test lists. Source
- SWE-bench uses Docker for reproducible evaluations. Source
- RFC 9457 defines problem details for HTTP APIs with type, title, status, detail and instance members and the application/problem+json media type. Source
- RFC 6585 defines the 429 Too Many Requests status code, which may include a Retry-After header. Source
- Stripe idempotency keys let clients retry safely; results are saved once execution begins, 500s are indeterminate, and requests rejected before execution do not cache a result. Source
See if your company qualifies
A short company assessment. No data uploads are needed.