Skip to content

Systems and records

Software cross-service debugging datasets: keep the trace evidence

By SourceX Editorial · Updated

9 sections

Short answer

Keep one bug record that joins the ticket, the distributed trace, and the commit, review and deployment for every service the fix touched. Store the trace id, each span's service name and version, and the commit deployed in each environment, so a reviewer can rebuild what ran, see which service failed first and confirm the fix in production instead of trusting a closed issue.

Record timeline

Software cross service

This page covers software cross-service debugging datasets: keep the trace evidence.Illustrative

Key takeaways

  • Key a cross-service bug record on the failing trace, because the service that raised the error is often not the one that caused it.
  • A closed issue is a repository event; GitHub closes it on merge to the default branch, which says nothing about deployment or outcome.
  • Trace id, span path, service.version, commit, change id and deployment environment are the identifiers that join the record.
  • Pin the version of every service on the failing path so a reviewer can rebuild the failure and test the fix in deployed order.
  • Mark broken chains from sampling, new traces at trust boundaries and queue hops instead of guessing the joins.
  • Trace headers must carry no personal data, but span attributes, logs and payloads often do, so screen them separately.

How do you preserve evidence for a software bug spanning several services?

Keep one bug record that is keyed on the distributed trace, not on the ticket. Store the trace id of at least one failing request, the service name and version of every span on the failing path, the commit and pull request that changed each service, and the deployment that put each change into production, then attach the production signal that shows the failure stopped.

The ticket is where people talked about the bug; the trace is where the bug actually happened. In a system of several services the service that raised the error is often not the one that caused it, and a record built from one repository's issue and patch can describe a fix to the symptom while the real change shipped somewhere else.

For a dataset, the difference decides whether a model can be tested on diagnosis. A linked record lets a reviewer ask which service failed first, which change fixed it and whether production confirms it. An unlinked one only lets a model guess at a plausible patch.

Why a closed issue in one repository is weak evidence for a cross-service fix

Because closing is a repository event, not a production event. GitHub's documentation says a linked issue closes automatically when the linked pull request merges into the repository's default branch, and that closing keywords can reference an issue in another repository with the owner and repository name. Nothing in that mechanism records whether the change was deployed, in which order relative to other services, or whether the error rate fell.

There is a second gap. GitHub also notes that when a closing keyword sits in a commit message rather than the pull request description, the issue closes on merge but the pull request is not listed as linked. A bug fixed across three repositories can therefore show one closed issue, one listed pull request and two changes that are only discoverable by searching commit messages.

Treat the closed state as one event in the record, with its own timestamp, and require separate evidence for deployment and for the production outcome.

Which identifiers join a trace to tickets, commits and deployments?

A small set of standard identifiers does most of the joining. W3C Trace Context defines the traceparent header, in which version 00 carries a 16-byte trace id written as 32 lowercase hex characters, an 8-byte parent id and trace flags; an all-zero trace id is invalid. OpenTelemetry's log data model defines optional TraceId and SpanId fields on log records, which is how an error log line points back to the request that produced it.

On the service side, OpenTelemetry's semantic conventions define service.name, service.namespace, service.instance.id and service.version, and note that the format of service.version is not defined by the conventions. Its version control attributes, currently at release-candidate stability, include vcs.ref.head.revision for the commit and vcs.change.id for the pull or merge request, and deployment.environment.name names the tier such as staging or production.

The link from code back to the ticket is the weakest join, because it depends on habit. Git supports trailers, key-value lines in a block at the end of a commit message, which give a ticket key a predictable place that tools can parse. Whatever convention a team uses, the record should store how each link was found: by an id the system wrote, by a key a person typed, or by a reviewer matching times and descriptions. Those three have very different error rates and should not be mixed silently.

Which identifiers join a trace to tickets, commits and deployments?
IdentifierDefined byWhere it usually livesWhat it joins
trace idW3C Trace Context traceparentTracing backend, request logsAll spans and logs of one request across services
span id and parent idW3C Trace Context, OpenTelemetryTracing backendCall order between services on the failing path
service.name and service.namespaceOpenTelemetry semantic conventionsSpan and log resource attributesA span to the team and repository that own it
service.versionOpenTelemetry semantic conventionsSpan resource attributes, build metadataA span to the build that was running
vcs.ref.head.revisionOpenTelemetry semantic conventionsBuild or deploy pipeline metadataA build to the commit it was made from
vcs.change.idOpenTelemetry semantic conventionsPipeline metadata, code hostA commit to its pull request and review
deployment.environment.nameOpenTelemetry semantic conventionsResource attributes, deploy logTelemetry to staging or production
Ticket key in a trailer or closing keywordGit trailers, code host conventionsCommit messages, pull request textA change back to the bug report

The distributed trace, ticket and code linkage specification

The specification below is the asset: one record per bug, with one child entry per service on the failing path. Fields marked required must be present for the record to count as linked; the others raise its value. Example values are illustrative.

At a glance

The distributed trace, ticket and code linkage specification

The distributed trace, ticket and code linkage specification
Field groupFieldRequiredValidation ruleIllustrative value
Bugticket key, opened time, reporter typeYesTicket exists and predates the first fix commitOPS-1182, customer-reported
Failing requesttrace id, failure time, entry serviceYes32 hex characters, not all zeros, inside the incident windowtrace id from a failing checkout request
Pathordered span list with service.name and parentYesEvery hop between entry and failing span is presentcheckout-api to pricing to inventory-sync
Running versionsservice.version per span at failure timeYesEach version maps to exactly one commitpricing at build 4.18.2
First faultservice and span judged to fail first, with reasonYesNamed by an engineer, with the evidence span citedinventory-sync timeout under retry load
Changesrepository, commit, change id, reviewer per serviceYesCommit is reachable from the deployed buildtwo repositories, two reviewed changes
Testsfailing test before fix, passing after, per servicePreferredTest fails on the old commit and passes on the new onecontract test on the pricing client
Deploymentsenvironment, time and version per serviceYesProduction deploy time follows the merge timepricing first, inventory-sync second
Outcomeproduction signal before and after, window lengthYesSame metric and window on both sideserror rate on the checkout route
Rights and screeningrepository owner, license, redaction logYesOwner confirmed; secrets and payloads removedinternal code, owned by the company

Can an outside reviewer rebuild every service on the path and confirm the fix?

Only if the record pins every service on the path, not just the one that was patched. A reviewer needs the commit each service was running at the time of failure, the configuration and feature flags that shaped the failing request, the dependency versions, and a test or replay that fails on the old combination and passes on the new one.

This is where a single-service packet falls short. If pricing was fixed but inventory-sync was redeployed the same day for an unrelated reason, the reviewer must be able to see that and check which change moved the outcome. Recording versions per span at failure time, and per deploy afterwards, is what makes that check possible.

The rebuild also tests the record itself. If a reviewer cannot get the old combination to fail, either a version is wrong, a configuration value is missing or the trace was not the failing request it was assumed to be. Each of those is worth knowing before the record is used to score a model, because a model that proposes the right fix should not be marked wrong against a broken answer key.

  • Check out each service at the recorded revision and confirm the build reports the recorded version.
  • Run the failing test or replay against the old set of revisions and confirm it fails the way the trace shows.
  • Apply the fix commits in the deployed order and confirm the failure stops at the expected step.
  • Compare the production outcome window against an unrelated deploy in the same period to rule out a coincidence.
  • Record the verdict as verified, partly verified or not reproducible, with the reason, rather than dropping the record.

Worked example (illustrative): a checkout timeout fixed in two other services

This example is illustrative; the services, counts and durations are invented. Customers report checkout failures, and a ticket is opened against the checkout API, which is where the timeout error appears. The trace for a failing request shows checkout-api calling pricing, which calls inventory-sync; inventory-sync is slow under load, and pricing's client retries three times without backoff, so the checkout call times out.

Two changes ship. Pricing adds backoff and a lower retry limit in one repository, and inventory-sync adds an index in another. The pricing pull request carries a closing keyword for the ticket, so the ticket closes when it merges, two days before the inventory-sync change reaches production. The record keeps both changes, their reviewers and their deploy times, and notes that the ticket closed before the fix was complete.

The outcome entry compares checkout errors for seven days before the first deploy with seven days after the second. A reviewer can see that errors fell partly after the pricing deploy and the rest after the inventory-sync deploy, so the first fault is recorded as inventory-sync latency, with the pricing retry policy as the amplifier. A model tested on this record can be scored on whether it looks downstream of the service that raised the error.

Where does the trace chain break, and how should the record show it?

Chains break at sampling decisions, trust boundaries and asynchronous hops, and the record should mark each break rather than paper over it. The W3C specification notes that recording only a subset of requests results in broken traces, and the sampled flag tells downstream services only whether the caller may have recorded trace data.

At a trust boundary a service may start a new trace instead of continuing the incoming one. OpenTelemetry describes span links for this case, and for batch or queue consumers whose span is started by many incoming items; links can point to spans in the same trace or in other traces. A record should keep those links as part of the path and flag any hop that had to be joined by time and payload instead of by id.

None of these breaks makes a record worthless. A path with one reconstructed hop, clearly marked, is still useful for training diagnosis; it is less useful as a verification case, where every join should be by id. Tag each record with a link quality value so a buyer can choose which records to use for which purpose.

  • Unsampled failing requests: keep the log lines with TraceId if present, and mark the path as reconstructed.
  • New trace at a boundary: store the link to the originating trace, not only the new trace id.
  • Queues and batches: record the producer span and the consumer span, joined by link or message id.
  • Missing service.version: mark the running version as unknown and exclude the record from verification tests.
  • Clock differences between hosts: order spans by parent relationship first and timestamps second.

What should be stripped from traces, logs and code before sharing?

Anything that identifies users or opens a door into the system, while keeping the identifiers that join the record. The W3C specification requires that traceparent and tracestate carry no personally identifiable information, but span attributes, URLs, log bodies and request payloads often do, so screen those fields separately.

  • Customer identifiers, emails, addresses and payment details inside span attributes, query strings and log bodies.
  • Secrets in configuration snapshots, environment variables, connection strings and test fixtures.
  • Internal hostnames and network addresses, replaced with stable pseudonyms so the path still reads correctly.
  • Third-party or vendored code whose license does not allow redistribution, which needs a code rights reviewer.
  • Repositories owned by a client or contractor rather than the company, which need the owner's permission.

Supplying linked engineering history for AI work

Bug records that join traces, tickets, reviewed changes and production outcomes across services are rarer than ticket exports, and they are the kind of evidence a model needs to learn diagnosis rather than patch writing. Before licensing them, confirm who owns each repository, what privacy rules apply to the telemetry and what customer contracts say; access to the systems does not settle any of those.

If an engineering organization wants to license such records, SourceX takes care of sourcing, rights review, contracting, preparation and delivery of the dataset. A short company assessment without any upload comes first, and large repositories and telemetry archives can remain in the company's own storage instead of being hosted by SourceX.

Frequently asked questions

Is the trace id enough to link a bug across services?

No. The trace id joins the spans of one request, but it does not say which build each service was running or which change fixed it. Add service.version per span, the commit behind each version and the deployment record.

What if our services do not report a version?

Then the running code at failure time cannot be pinned, and the record should mark the version as unknown. Many teams add the build commit or tag to the service's resource attributes in the deploy pipeline so future records carry it.

Should a bug record include the full trace?

Keep the failing path and the spans that explain it, not every span in a large trace. Unrelated spans add volume and more personal data to screen without adding evidence.

How do we link commits in several repositories to one ticket?

Use the ticket key in a consistent place in every commit or pull request, such as a commit trailer or the pull request description. Then check the record by searching every repository on the failing path for that key.

What makes a cross-service bug record unusable?

A missing trace or log link to the failure, no version for the failing services, or no production signal after the fix. Records with one of these gaps can still be kept but should be labeled partial.

Sources

  • The W3C Trace Context traceparent header carries a 16-byte trace id, an 8-byte parent id and trace flags including a sampled flag, and notes that recording only a subset of requests breaks traces. Source
  • W3C Trace Context requires that traceparent and tracestate carry no personally identifiable information. Source
  • OpenTelemetry span links connect causally related spans in the same or different traces, including a new trace started at a trust boundary and batch processing. Source
  • OpenTelemetry's log data model defines optional TraceId and SpanId fields so log records can point to the request and span that produced them. Source
  • OpenTelemetry semantic conventions define service.name, service.namespace, service.instance.id and service.version, and leave the version format undefined. Source
  • OpenTelemetry version control attributes include vcs.ref.head.revision for the commit and vcs.change.id for the pull or merge request. Source
  • OpenTelemetry's deployment.environment.name attribute names the deployment tier, such as staging or production. Source
  • GitHub closes a linked issue when the linked pull request merges into the default branch, supports cross-repository closing keywords, and does not list a pull request as linked when the keyword is only in a commit message. Source
  • Git documents trailers as key-value pairs in a block at the end of a commit message. Source

See if your company qualifies

A short company assessment. No data uploads are needed.

See if your company qualifies