Systems and records
Software cross-service debugging datasets: keep the trace evidence
By SourceX Editorial · Updated
9 sections
Short answer
Keep one bug record that joins the ticket, the distributed trace, and the commit, review and deployment for every service the fix touched. Store the trace id, each span's service name and version, and the commit deployed in each environment, so a reviewer can rebuild what ran, see which service failed first and confirm the fix in production instead of trusting a closed issue.
Record timeline
Software cross service
Key takeaways
- Key a cross-service bug record on the failing trace, because the service that raised the error is often not the one that caused it.
- A closed issue is a repository event; GitHub closes it on merge to the default branch, which says nothing about deployment or outcome.
- Trace id, span path, service.version, commit, change id and deployment environment are the identifiers that join the record.
- Pin the version of every service on the failing path so a reviewer can rebuild the failure and test the fix in deployed order.
- Mark broken chains from sampling, new traces at trust boundaries and queue hops instead of guessing the joins.
- Trace headers must carry no personal data, but span attributes, logs and payloads often do, so screen them separately.
How do you preserve evidence for a software bug spanning several services?
Keep one bug record that is keyed on the distributed trace, not on the ticket. Store the trace id of at least one failing request, the service name and version of every span on the failing path, the commit and pull request that changed each service, and the deployment that put each change into production, then attach the production signal that shows the failure stopped.
The ticket is where people talked about the bug; the trace is where the bug actually happened. In a system of several services the service that raised the error is often not the one that caused it, and a record built from one repository's issue and patch can describe a fix to the symptom while the real change shipped somewhere else.
For a dataset, the difference decides whether a model can be tested on diagnosis. A linked record lets a reviewer ask which service failed first, which change fixed it and whether production confirms it. An unlinked one only lets a model guess at a plausible patch.
Why a closed issue in one repository is weak evidence for a cross-service fix
Because closing is a repository event, not a production event. GitHub's documentation says a linked issue closes automatically when the linked pull request merges into the repository's default branch, and that closing keywords can reference an issue in another repository with the owner and repository name. Nothing in that mechanism records whether the change was deployed, in which order relative to other services, or whether the error rate fell.
There is a second gap. GitHub also notes that when a closing keyword sits in a commit message rather than the pull request description, the issue closes on merge but the pull request is not listed as linked. A bug fixed across three repositories can therefore show one closed issue, one listed pull request and two changes that are only discoverable by searching commit messages.
Treat the closed state as one event in the record, with its own timestamp, and require separate evidence for deployment and for the production outcome.
Which identifiers join a trace to tickets, commits and deployments?
A small set of standard identifiers does most of the joining. W3C Trace Context defines the traceparent header, in which version 00 carries a 16-byte trace id written as 32 lowercase hex characters, an 8-byte parent id and trace flags; an all-zero trace id is invalid. OpenTelemetry's log data model defines optional TraceId and SpanId fields on log records, which is how an error log line points back to the request that produced it.
On the service side, OpenTelemetry's semantic conventions define service.name, service.namespace, service.instance.id and service.version, and note that the format of service.version is not defined by the conventions. Its version control attributes, currently at release-candidate stability, include vcs.ref.head.revision for the commit and vcs.change.id for the pull or merge request, and deployment.environment.name names the tier such as staging or production.
The link from code back to the ticket is the weakest join, because it depends on habit. Git supports trailers, key-value lines in a block at the end of a commit message, which give a ticket key a predictable place that tools can parse. Whatever convention a team uses, the record should store how each link was found: by an id the system wrote, by a key a person typed, or by a reviewer matching times and descriptions. Those three have very different error rates and should not be mixed silently.
| Identifier | Defined by | Where it usually lives | What it joins |
|---|---|---|---|
| trace id | W3C Trace Context traceparent | Tracing backend, request logs | All spans and logs of one request across services |
| span id and parent id | W3C Trace Context, OpenTelemetry | Tracing backend | Call order between services on the failing path |
| service.name and service.namespace | OpenTelemetry semantic conventions | Span and log resource attributes | A span to the team and repository that own it |
| service.version | OpenTelemetry semantic conventions | Span resource attributes, build metadata | A span to the build that was running |
| vcs.ref.head.revision | OpenTelemetry semantic conventions | Build or deploy pipeline metadata | A build to the commit it was made from |
| vcs.change.id | OpenTelemetry semantic conventions | Pipeline metadata, code host | A commit to its pull request and review |
| deployment.environment.name | OpenTelemetry semantic conventions | Resource attributes, deploy log | Telemetry to staging or production |
| Ticket key in a trailer or closing keyword | Git trailers, code host conventions | Commit messages, pull request text | A change back to the bug report |
The distributed trace, ticket and code linkage specification
The specification below is the asset: one record per bug, with one child entry per service on the failing path. Fields marked required must be present for the record to count as linked; the others raise its value. Example values are illustrative.
At a glance
The distributed trace, ticket and code linkage specification
| Field group | Field | Required | Validation rule | Illustrative value |
|---|---|---|---|---|
| Bug | ticket key, opened time, reporter type | Yes | Ticket exists and predates the first fix commit | OPS-1182, customer-reported |
| Failing request | trace id, failure time, entry service | Yes | 32 hex characters, not all zeros, inside the incident window | trace id from a failing checkout request |
| Path | ordered span list with service.name and parent | Yes | Every hop between entry and failing span is present | checkout-api to pricing to inventory-sync |
| Running versions | service.version per span at failure time | Yes | Each version maps to exactly one commit | pricing at build 4.18.2 |
| First fault | service and span judged to fail first, with reason | Yes | Named by an engineer, with the evidence span cited | inventory-sync timeout under retry load |
| Changes | repository, commit, change id, reviewer per service | Yes | Commit is reachable from the deployed build | two repositories, two reviewed changes |
| Tests | failing test before fix, passing after, per service | Preferred | Test fails on the old commit and passes on the new one | contract test on the pricing client |
| Deployments | environment, time and version per service | Yes | Production deploy time follows the merge time | pricing first, inventory-sync second |
| Outcome | production signal before and after, window length | Yes | Same metric and window on both sides | error rate on the checkout route |
| Rights and screening | repository owner, license, redaction log | Yes | Owner confirmed; secrets and payloads removed | internal code, owned by the company |
Can an outside reviewer rebuild every service on the path and confirm the fix?
Only if the record pins every service on the path, not just the one that was patched. A reviewer needs the commit each service was running at the time of failure, the configuration and feature flags that shaped the failing request, the dependency versions, and a test or replay that fails on the old combination and passes on the new one.
This is where a single-service packet falls short. If pricing was fixed but inventory-sync was redeployed the same day for an unrelated reason, the reviewer must be able to see that and check which change moved the outcome. Recording versions per span at failure time, and per deploy afterwards, is what makes that check possible.
The rebuild also tests the record itself. If a reviewer cannot get the old combination to fail, either a version is wrong, a configuration value is missing or the trace was not the failing request it was assumed to be. Each of those is worth knowing before the record is used to score a model, because a model that proposes the right fix should not be marked wrong against a broken answer key.
- Check out each service at the recorded revision and confirm the build reports the recorded version.
- Run the failing test or replay against the old set of revisions and confirm it fails the way the trace shows.
- Apply the fix commits in the deployed order and confirm the failure stops at the expected step.
- Compare the production outcome window against an unrelated deploy in the same period to rule out a coincidence.
- Record the verdict as verified, partly verified or not reproducible, with the reason, rather than dropping the record.
Worked example (illustrative): a checkout timeout fixed in two other services
This example is illustrative; the services, counts and durations are invented. Customers report checkout failures, and a ticket is opened against the checkout API, which is where the timeout error appears. The trace for a failing request shows checkout-api calling pricing, which calls inventory-sync; inventory-sync is slow under load, and pricing's client retries three times without backoff, so the checkout call times out.
Two changes ship. Pricing adds backoff and a lower retry limit in one repository, and inventory-sync adds an index in another. The pricing pull request carries a closing keyword for the ticket, so the ticket closes when it merges, two days before the inventory-sync change reaches production. The record keeps both changes, their reviewers and their deploy times, and notes that the ticket closed before the fix was complete.
The outcome entry compares checkout errors for seven days before the first deploy with seven days after the second. A reviewer can see that errors fell partly after the pricing deploy and the rest after the inventory-sync deploy, so the first fault is recorded as inventory-sync latency, with the pricing retry policy as the amplifier. A model tested on this record can be scored on whether it looks downstream of the service that raised the error.
Where does the trace chain break, and how should the record show it?
Chains break at sampling decisions, trust boundaries and asynchronous hops, and the record should mark each break rather than paper over it. The W3C specification notes that recording only a subset of requests results in broken traces, and the sampled flag tells downstream services only whether the caller may have recorded trace data.
At a trust boundary a service may start a new trace instead of continuing the incoming one. OpenTelemetry describes span links for this case, and for batch or queue consumers whose span is started by many incoming items; links can point to spans in the same trace or in other traces. A record should keep those links as part of the path and flag any hop that had to be joined by time and payload instead of by id.
None of these breaks makes a record worthless. A path with one reconstructed hop, clearly marked, is still useful for training diagnosis; it is less useful as a verification case, where every join should be by id. Tag each record with a link quality value so a buyer can choose which records to use for which purpose.
- Unsampled failing requests: keep the log lines with TraceId if present, and mark the path as reconstructed.
- New trace at a boundary: store the link to the originating trace, not only the new trace id.
- Queues and batches: record the producer span and the consumer span, joined by link or message id.
- Missing service.version: mark the running version as unknown and exclude the record from verification tests.
- Clock differences between hosts: order spans by parent relationship first and timestamps second.
What should be stripped from traces, logs and code before sharing?
Anything that identifies users or opens a door into the system, while keeping the identifiers that join the record. The W3C specification requires that traceparent and tracestate carry no personally identifiable information, but span attributes, URLs, log bodies and request payloads often do, so screen those fields separately.
- Customer identifiers, emails, addresses and payment details inside span attributes, query strings and log bodies.
- Secrets in configuration snapshots, environment variables, connection strings and test fixtures.
- Internal hostnames and network addresses, replaced with stable pseudonyms so the path still reads correctly.
- Third-party or vendored code whose license does not allow redistribution, which needs a code rights reviewer.
- Repositories owned by a client or contractor rather than the company, which need the owner's permission.
Supplying linked engineering history for AI work
Bug records that join traces, tickets, reviewed changes and production outcomes across services are rarer than ticket exports, and they are the kind of evidence a model needs to learn diagnosis rather than patch writing. Before licensing them, confirm who owns each repository, what privacy rules apply to the telemetry and what customer contracts say; access to the systems does not settle any of those.
If an engineering organization wants to license such records, SourceX takes care of sourcing, rights review, contracting, preparation and delivery of the dataset. A short company assessment without any upload comes first, and large repositories and telemetry archives can remain in the company's own storage instead of being hosted by SourceX.
Frequently asked questions
Is the trace id enough to link a bug across services?
No. The trace id joins the spans of one request, but it does not say which build each service was running or which change fixed it. Add service.version per span, the commit behind each version and the deployment record.
What if our services do not report a version?
Then the running code at failure time cannot be pinned, and the record should mark the version as unknown. Many teams add the build commit or tag to the service's resource attributes in the deploy pipeline so future records carry it.
Should a bug record include the full trace?
Keep the failing path and the spans that explain it, not every span in a large trace. Unrelated spans add volume and more personal data to screen without adding evidence.
How do we link commits in several repositories to one ticket?
Use the ticket key in a consistent place in every commit or pull request, such as a commit trailer or the pull request description. Then check the record by searching every repository on the failing path for that key.
What makes a cross-service bug record unusable?
A missing trace or log link to the failure, no version for the failing services, or no production signal after the fix. Records with one of these gaps can still be kept but should be labeled partial.
Sources
- The W3C Trace Context traceparent header carries a 16-byte trace id, an 8-byte parent id and trace flags including a sampled flag, and notes that recording only a subset of requests breaks traces. Source
- W3C Trace Context requires that traceparent and tracestate carry no personally identifiable information. Source
- OpenTelemetry span links connect causally related spans in the same or different traces, including a new trace started at a trust boundary and batch processing. Source
- OpenTelemetry's log data model defines optional TraceId and SpanId fields so log records can point to the request and span that produced them. Source
- OpenTelemetry semantic conventions define service.name, service.namespace, service.instance.id and service.version, and leave the version format undefined. Source
- OpenTelemetry version control attributes include vcs.ref.head.revision for the commit and vcs.change.id for the pull or merge request. Source
- OpenTelemetry's deployment.environment.name attribute names the deployment tier, such as staging or production. Source
- GitHub closes a linked issue when the linked pull request merges into the default branch, supports cross-repository closing keywords, and does not list a pull request as linked when the keyword is only in a commit message. Source
- Git documents trailers as key-value pairs in a block at the end of a commit message. Source
See if your company qualifies
A short company assessment. No data uploads are needed.