Packaging Speech Datasets: Manifests, Segment Timing and Diarization Files

This page covers packaging and linking. Sample rate, bit depth and codecs are covered in audio file specs for speech datasets, and the wider family of delivery formats sits under the dataset delivery formats hub.

What a speech delivery package should contain

A speech package should contain untouched source audio, derived segment-level manifests, speaker-turn files and the documentation needed to interpret them, all addressed by the same recording ID. Keep each layer in its own folder so a buyer can regenerate segments without touching the originals. The recording ID (for example rec_000184) should be the join key everywhere: file name stem, manifest field, RTTM file ID column and metadata table.

Illustrative example: invented to show structure; it does not describe an available dataset.

delivery_v1.2/
  bagit.txt
  manifest-sha256.txt          # checksum of every payload file
  data/
    audio/rec_000184.wav       # original 2-channel recording
    manifests/train.jsonl      # one utterance per line
    manifests/dev.jsonl
    manifests/test.jsonl
    rttm/rec_000184.rttm       # speaker turns, whole recording
    uem/rec_000184.uem         # scored regions
    metadata/recordings.parquet  # one row per recording
    metadata/channel_map.csv
    docs/CONVENTIONS.md        # normalization, tags, timing rules
    docs/CHANGELOG.md

The outer wrapper here follows BagIt (RFC 8493), which pairs a payload directory with checksum manifests so the receiving team can verify every byte [6]. For per-file hashing practice see dataset manifests and checksums. A Croissant JSON-LD file can sit beside the bag to describe the file resources and record fields in a machine-readable way [7]; the Croissant metadata guide explains what to request.

The utterance manifest: fields and format

The utterance manifest is a JSON Lines file with one object per segment, and its minimum useful fields are an audio path, start offset, duration and transcript. Open-source ASR toolkits such as NVIDIA NeMo use this shape, with keys like audio_filepath, text and duration in seconds and one manifest per train, dev and test split; confirm exact key names against the toolkit version you run. JSON Lines itself requires UTF-8 with no byte order mark and a valid JSON value on every line, so a blank trailing line or a Windows BOM will break strict readers [1].

For long recordings, ask for segments expressed as an offset plus duration into the original file rather than thousands of pre-cut clips. Many loaders accept an offset key, but confirm the field name and units against the toolkit version your team actually runs. Pre-cut clips are still reasonable for short dictation; record which approach was used.

Illustrative example: invented to show structure; it does not describe an available dataset.

{"utt_id": "rec_000184_0007", "recording_id": "rec_000184", "audio_filepath": "audio/rec_000184.wav", "offset": 41.280, "duration": 6.415, "channel": 1, "speaker": "rec_000184_spk_customer", "role": "customer", "text": "yeah i was charged twice on the september statement", "text_raw": "Yeah, I was charged twice on the September statement.", "language": "en-US", "redaction": [{"start": 44.900, "end": 45.700, "type": "account_number"}], "split": "train"}

Useful additions beyond the toolkit minimum:

• utt_id and recording_id, so segments map back to recordings and to RTTM rows.
• channel, using a documented index base (0 or 1, stated once in the conventions file).
• speaker as a recording-scoped pseudonymous ID plus a role (agent, customer, IVR, unknown).
• Both a normalized text for training and a text_raw with casing and punctuation.
• Redaction spans in seconds, so masked audio and bracketed transcript tokens line up; see redacting spoken PII from call recordings.

Segment timing and alignment precision

Segment timing should be stated in seconds from the start of the original file, with a declared precision and a declared method of derivation. Three different timing sources show up in real deliveries: human-marked boundaries in an annotation tool, timestamps exported by a contact-center speech analytics engine, and forced alignment run after transcription. They have different error profiles, so a timing_source field per segment (or per recording) is worth requesting.

Agree on precision up front. Millisecond resolution (three decimals) is common in manifests and RTTM, but a value written to the millisecond is not necessarily accurate to the millisecond. Ask the supplier to state the expected boundary tolerance and whether segments were padded with silence. For word-level alignment, request a separate file (CTM-style word rows or a JSONL of word, start, end, confidence) rather than packing words into the utterance manifest.

Common timing failure modes to test on a sample:

• Offsets computed against a resampled or trimmed file, not the delivered original.
• Clock drift between a channel-split file and a mixed-down file of the same call.
• Segments that overlap on the same channel because of crosstalk handling, with no flag.
• Durations that run past the end of the audio, which most loaders reject or silently truncate.

Wall-clock time of the call belongs in recording metadata, not segment offsets. The timestamps and time zones guide covers ISO 8601 and ordering rules for those fields.

RTTM and UEM files for diarization labels

RTTM is the de facto plain-text format for speaker turns: one turn per line, space-delimited, ten fields beginning with SPEAKER. The fields are commonly given as type, file ID, channel, turn onset, turn duration, orthography, speaker type, speaker name, confidence and signal lookahead time, with unused fields written as <NA> and speaker names unique within each file. Diarization challenge organizers have required start time, duration and speaker ID at fixed column positions, so treat column order as strict [2]. Toolkits typically pair RTTM with UEM files that list scored regions as file ID, channel, start and end.

Illustrative example: invented to show structure; it does not describe an available dataset.

SPEAKER rec_000184 1 41.280 6.415 <NA> <NA> rec_000184_spk_customer <NA> <NA>
SPEAKER rec_000184 1 47.102 1.930 <NA> <NA> rec_000184_spk_agent <NA> <NA>
SPEAKER rec_000184 1 48.650 0.540 <NA> <NA> rec_000184_spk_customer <NA> <NA>

In this example the RTTM indexes the mixed-down recording, so every turn uses channel 1, the value most scoring tools expect; the per-leg mapping lives in channel_map.csv. Overlapping speech is represented by separate lines whose time ranges overlap, which is exactly what diarization error rate scoring needs. RTTM was defined in NIST's Rich Transcription evaluations; verify field details against the NIST plan or your scoring tool before publishing internal specs, since secondary documentation varies.

Packaging decisions to state explicitly:

• Whether RTTM refers to the mixed-down file or the stereo original, and which channel number it uses.
• Whether speaker names match the manifest speaker values exactly (they should).
• How non-speech (hold music, IVR prompts, beeps) is excluded: via UEM regions or by omitting turns.
• Whether turns were derived from channel separation or human annotation, because channel-derived turns miss same-side crosstalk.

See the diarization glossary entry for the underlying concepts.

Stereo call recordings: keep, split or both

For contact-center audio, ask for the original two-channel file and a channel map, and treat split mono files as a derived convenience. Telephony platforms commonly record each call leg to its own channel: as of October 2026, Twilio's RecordingChannels parameter accepts mono or dual, defaults to mono, and in dual mode keeps each leg of a two-party call in a separate channel [3]. Amazon Connect lets administrators choose to record the customer, the agent or both [4], and speech services can label transcripts by channel when speakers are separated this way [5].

The trap is assuming which side is left. Sources and configurations disagree about which party lands on channel 0, and transfers, conferences and IVR legs complicate it further. Require a channel_map.csv with one row per recording rather than a single global rule.

Illustrative example: invented to show structure; it does not describe an available dataset.

If the supplier also delivers split mono files, they should share sample-accurate timing with the stereo original, and manifest offsets should name which file they index. Mono-only archives lose this information permanently, which matters for both ASR and diarization; the trade-offs are covered in dual-channel call recordings for ASR and diarization.

Transcript conventions document

The conventions document is the contract for how text maps to sound, and without it two batches from the same supplier can be silently incompatible. It should be versioned alongside the data and referenced from the changelog. Ask for these items in it:

• Normalization rules: casing, punctuation, numbers (verbatim "twenty three" vs "23"), dates, currency, acronyms and spelling variants.
• Disfluency policy: whether fillers, false starts, repetitions and partial words are transcribed and how they are marked.
• Non-speech tags: a closed list such as [noise], [music], [laughter], [crosstalk], [inaudible].
• Redaction tokens: exact strings (for example [ACCOUNT_NUMBER]) and how their timing is recorded.
• Code-switching and language tags, including the locale codes used.
• Timing rules: precision, padding, minimum and maximum segment length, and how overlaps are segmented.
• Transcription provenance: human, machine with human review, or machine only, at segment level where it varies.

Pre-delivery checklist for speech packages

A short automated check catches most packaging defects before any model training run. Run these on the sample and again on each full delivery.

Illustrative example: invented to show structure; it does not describe an available dataset.

Speaker leakage across splits is a common silent failure: when the same agent appears in train and test, word error rates can look better than production. Split by recording at minimum, and by agent where agent IDs are available in pseudonymous form.

How SourceX handles speech data requests

SourceX sources operational datasets from US companies on request, including support and sales histories and new recordings of hands-on work, and manages the licensing process; nothing is held in stock and a request does not guarantee a match. Every dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, through private, access-controlled workflows after an executed agreement and supplier approval. Personal details such as names, phone and account numbers are removed or replaced before delivery, with the method recorded and a sample checked, though no method is perfect. You can describe the manifest fields, channel layout and diarization files you need when you submit a buyer request; related context is on the call center audio datasets page and voice and audio data licensing.

Request licensed speech data with the packaging you need

Describe the recordings, transcript conventions, manifest fields and diarization labels your ASR or voice-agent team needs, and SourceX looks for US businesses that hold that data. Each release is approved by the supplying company, and pricing and allowed uses are agreed in a license before anything is delivered. Start at sourcex.si/buyers.