Systems and records
Parquet or JSONL for a dataset? Decide by the buyer's workflow
By SourceX Editorial · Updated
10 sections
Short answer
Choose by the first job the buyer runs on the delivery. Use JSONL when each record is one self-contained task, such as a conversation, a ticket thread or an agent trajectory read whole by a training or evaluation loader. Use Parquet when the buyer will filter, sample, join or profile many fixed fields across millions of rows. Either way, ship string IDs, a manifest and a validator.
Value drivers
Dataset Parquet JSONL
Key takeaways
- Pick the format for the buyer's first job: whole-record reads favor JSONL, column scans, filters and joins favor Parquet.
- Link tasks, steps and artifacts by stable string IDs, never by row order or file position.
- Write IDs as strings, because JSON parsers agree on exact integers only up to about nine quadrillion.
- Ship a package contract with schema and content versions, a manifest with hashes and a validator that needs no supplier systems.
- If both formats are needed, make one canonical and generate the other with a checked, shipped script.
When should a business task dataset use Parquet versus JSONL?
Ask what the buyer does first with the files, not which format is better. If the first job reads each record whole, one task at a time, JSONL fits because one line is one task. If the first job scans a few columns across a large table to filter, sample, deduplicate or join, Parquet fits because it can read only the columns needed.
Business task data makes the choice harder than a flat table would. A support case, an invoice exception or a CRM workflow has a parent task, a variable number of steps and links to artifacts such as attachments, emails and system records. The format has to keep those links intact for the buyer's workflow while staying easy to validate.
The decision matrix below maps eight common buyer workflows to a first-choice format, the reason, what the package must include and the condition that flips the choice.
What does each format actually specify?
JSON Lines is a text convention with three rules: the file is UTF-8, each line is one valid JSON value, and lines end with a newline character. The jsonlines.org specification recommends a newline after the last value because it makes generating and concatenating files easier, suggests the .jsonl extension, and recommends stream compressors such as gzip for saving space. It says nothing about schema, types or required fields; those belong to your package contract.
Parquet is a binary columnar file format. Its specification describes a file as one or more row groups, each holding one column chunk per column, with column chunks divided into pages that are the unit of encoding and compression. File metadata is written after the data, and readers are expected to read that metadata first to find the column chunks they need. Parquet supports nested structures using the record shredding approach from the Dremel paper, compression chosen per column, and min and max statistics at several levels.
Those design facts drive the practical differences. Apache Arrow's documentation notes that reading a subset of columns can be much faster than reading a whole file because of the columnar layout, and Arrow's dataset reader can filter on any column using row group statistics. A JSONL reader has to parse every line to find anything.
Neither format defines meaning. Parquet enforces that a column holds integers or strings, but not that status_code 7 means closed, that a timestamp is UTC or that an amount is in cents. JSONL enforces even less: two lines can disagree on the type of the same field and both are valid JSON Lines. Units, time zones, enumerations and the relationship between records belong in the data dictionary and the validator, whichever container carries them.
Which format fits each buyer workflow?
Read across the row for the buyer's first job. The switch condition is the assumption that, if it turns out false, moves the recommendation to the other format.
Three rows cause most of the debate. Evaluation replay favors JSONL because an agent or workflow task is read as a unit: the harness needs the instruction, starting state, allowed tools, step history and expected end state together, and splitting them across tables means every replay begins with a join. Filtering and sampling favor Parquet because the selection reads a handful of fields, such as product area, outcome and date, across every task, and a columnar reader never touches the long text fields while doing it. Recurring appends are a draw: JSONL files can simply be extended, while Parquet deliveries add files per batch, which suits buyers whose lake already organizes data that way.
The matrix is a starting point, not a verdict. Two buyers with the same records can reasonably choose differently because their first jobs differ, and the agreement should record the choice and the reason so a later delivery does not change format by accident.
| Buyer's first job | Start with | Why | Switch if |
|---|---|---|---|
| Supervised fine-tuning on conversations or instructions | JSONL | Trainers commonly expect one conversation per record, such as a messages list of role and content turns | The corpus is very large and the buyer streams it from object storage; Parquet with a list of structs streams as well |
| Evaluation replay of agent or workflow tasks | JSONL per task | The harness loads one task with its steps, tools and expected end state as a unit | Hundreds of thousands of tasks need slicing by app, outcome or date before replay |
| Filtering, stratified sampling and deduplication | Parquet | Only the filter and key columns are read, and row group statistics skip irrelevant data | The buyer has no columnar tooling and the set is small enough to scan |
| Joining tasks to warehouse tables or other deliveries | Parquet | Typed columns and stable keys load directly into query engines | Join keys live inside nested step records that the buyer will not flatten |
| Rights, privacy and quality review by people | JSONL sample | A reviewer can open, search and comment on one record without code | Reviewers already work in a notebook or SQL tool |
| Profiling distributions and drift across deliveries | Parquet | Column statistics and fast column reads make per-field profiles cheap | Fields are mostly free text whose structure varies by record |
| Recurring appends from a live source | JSONL or new Parquet files per batch | JSONL files concatenate; a finished Parquet file is closed by its footer, so each batch arrives as new files | The buyer's lake already partitions Parquet by date |
| Pretraining-style text at large scale | Parquet | Compression per column and column pruning keep text and metadata reads separate | The loader in the buyer's stack only reads line-delimited text |
How do tasks stay connected to their artifacts in either format?
Give every task, step and artifact a stable string ID and link them by ID, never by row position or file order. Row order is not a contract in either format, and files get resharded, filtered and merged downstream.
Write IDs as strings even when the source system uses integers. RFC 8259, the JSON standard, notes that integers are interoperable across implementations only within the range from minus two to the 53rd power plus one to two to the 53rd power minus one; larger numeric IDs can be silently rounded by a JSON parser. Parquet can store 64-bit integers exactly, but a JSONL review sample of the same data would not, so strings keep both formats in agreement.
Keep large binary artifacts, such as PDFs, screenshots and audio, as separate files referenced by relative path and hash. Embedding them in a JSONL line as base64 inflates every read, and embedding them in a Parquet column makes column statistics meaningless for that field.
- tasks: one row per task with task_id, workflow type, outcome, timestamps and source system.
- steps: one row per step with step_id, task_id, order, actor role, action and resulting state, nested inside the task in JSONL or as its own table in Parquet.
- artifacts: one row per file with artifact_id, task_id, relative path, media type, byte size and SHA-256.
- Every reference resolves inside the package; nothing points to a supplier URL or internal share.
Can a buyer validate and rebuild the package without your internal systems?
Only if the package carries its own contract. The buyer should be able to run one validator against the delivered files and get a pass or a list of failures, without a call to the supplier and without access to the systems the data came from.
The checklist below applies to both formats; only the mechanism differs.
A good validator runs in a fixed order and stops on the first class of failure. It recomputes hashes and counts against the manifest, loads every file with the declared schema, checks uniqueness of IDs and resolution of every foreign key, verifies that each artifact path exists, and finally checks enumerations, units and time zones against the data dictionary. Shipping its report with the delivery means the buyer reruns the same checks and compares results instead of starting an investigation.
| Package element | In a Parquet delivery | In a JSONL delivery | Validator check |
|---|---|---|---|
| Schema version | Arrow or Parquet schema file plus a version string in the manifest | JSON Schema file plus a version string in the manifest | Every file declares the version it was written with |
| Content version | Delivery ID and date in the manifest | Delivery ID and date in the manifest | Version matches the delivery note and the agreement schedule |
| Stable IDs | String columns with uniqueness stated per table | String fields with uniqueness stated per record type | No duplicates; every foreign key resolves |
| Manifest | List of files with row counts and SHA-256 | List of files with line counts and SHA-256 | Counts and hashes recomputed and matched |
| Relative paths | Artifact paths relative to the package root | Same | Every path exists inside the package |
| Field semantics | Data dictionary: meaning, unit, time zone, allowed values | Same dictionary | Enumerations and units checked against the dictionary |
| Roles | Actor role column with a closed list | Role field on each turn or step | No role outside the list; no personal names in role fields |
| Runnable validator | Script that reads only the delivered files | Same script | Exit code and report shipped with the delivery |
Which format wins when one support history serves two buyer jobs? An illustrative scenario
This scenario is invented to show the decision. A software company licenses de-identified support history: cases, the message turns inside them, internal notes, status changes and attachments. The buyer has two jobs: fine-tune a support assistant on resolved conversations, and first build an evaluation slice stratified by product area, severity and outcome.
The first job the buyer runs is the stratified selection, which reads 4 columns across all cases and joins to a product table. That favors Parquet as the canonical delivery: a cases table, a turns table and an artifacts table, linked by string IDs. The fine-tuning job then needs one conversation per record in a messages format, which the buyer, or the supplier as a derived file, produces from the turns table.
The deciding check is whether the derived JSONL can be rebuilt exactly from the canonical tables. The validator regenerates it, compares case counts and per-case turn counts, and hashes the output. In this illustration, 100 percent of cases in the derived file must match the canonical tables; any mismatch blocks the delivery rather than being noted.
The exception: if the buyer had only wanted fine-tuning on a few thousand conversations, the canonical Parquet tables would add work without benefit. A single JSONL file with a published schema and a manifest would be the better package.
Either way, the choice is recorded in the agreement schedule with the reason, the canonical format named, and the derived file listed as generated, so a later delivery cannot quietly switch format or let the two drift apart.
Which workflow assumption would flip the format choice?
Write the assumptions into the delivery note so the buyer can see which one their workflow violates. Most format disputes come from an assumption nobody stated, not from the format itself.
Steps
Which workflow assumption would flip the format choice
- Size: below the point where a full scan is cheap for the buyer, Parquet's column pruning saves little and JSONL's readability wins.
- Access pattern: whole-record reads favor JSONL; column scans, filters and joins favor Parquet.
- Nesting: deep, irregular structures are natural in JSON; in Parquet they need an explicit nested schema that every writer follows.
- Schema stability: a field set that changes every delivery is easier to version in JSONL with JSON Schema; a stable field set benefits from Parquet's typed columns.
- Tooling: Hugging Face datasets loads both JSON and Parquet files and can stream a dataset without downloading it first, so many training stacks accept either; confirm the buyer's actual loader before deciding.
- Review: if people outside engineering must read records, ship a JSONL sample whatever the canonical format is.
How large does a dataset need to be before Parquet pays off?
There is no universal row count, so measure it in the buyer's own stack rather than trusting a rule of thumb. The gain from Parquet comes from skipping columns and row groups the job does not need, which depends on how many fields the job reads, how wide the text fields are and which engine does the reading.
A fair test takes a representative sample of the actual delivery, writes it in both formats with the compression the delivery would use, and times the buyer's real first job on each: the selection query, the loader or the profiling script. Record the engine and version, the sample size and the hardware, and repeat each run several times so one slow read does not decide the question.
If the difference on the sample is small, prefer the format that is easier to validate and review. If it is large and grows with sample size, the full delivery will show the same pattern, and Parquet earns its extra tooling.
When is shipping both formats worth the extra work?
Ship both only when the buyer's workflow genuinely has two first jobs, and then make one canonical and the other derived. Two independently produced files drift: a fix applied to one is forgotten in the other, and the buyer cannot tell which is right.
The rule is simple. Name the canonical format in the agreement schedule, generate the derived files with a script included in the package, and make equivalence a validator check with counts and hashes. If the derived format is only for human review, cap it as a sample and say so in the manifest.
Asking for licensed business task data in the format your workflow needs
Write the request around the work, not the file extension. A sentence such as first we select an evaluation slice by product area and outcome, then we fine-tune on resolved conversations tells a supplier more than a bare request for Parquet, and it lets the supplier flag early when the source system cannot produce a field the workflow depends on.
Put the matrix row that matches your first job into the data request, along with the package elements you expect to validate. Format is easy to change before preparation starts and expensive to change after a delivery has been reviewed and signed off.
Because SourceX looks for each dataset at U.S. businesses only after a buyer asks, the request form, which covers scope, volume, language, format, timing and licensing needs, is the place to state the canonical format and package contract from this page.
Frequently asked questions
Is JSONL or Parquet better for fine-tuning data?
For most fine-tuning on conversations, JSONL is the simpler fit because each line holds one complete example, such as a messages list of role and content turns. Parquet also works and is often used for very large corpora. Check what the trainer's loader expects before deciding.
Can Parquet store nested conversations and agent steps?
Yes. Parquet was designed for nested data and uses the record shredding approach described in the Dremel paper. A conversation can be a list of structs in one column, though every writer has to follow the same nested schema.
Why should IDs be strings in a JSONL delivery?
Because JSON parsers commonly use double-precision numbers. RFC 8259 notes that only integers within plus or minus two to the 53rd power, less one, are read identically by all implementations, so larger numeric IDs can be rounded without warning.
Should a dataset be delivered in both Parquet and JSONL?
Only when the buyer has two distinct first jobs. Make one format canonical, generate the other with a script shipped in the package, and check equivalence with counts and hashes so the two cannot drift.
Can a JSONL file be appended to, and a Parquet file?
A JSONL file can be extended by adding lines, and files concatenate when each line ends with a newline. A Parquet file ends with its metadata footer, so new data normally arrives as new Parquet files that the reader treats as one dataset.
Sources
- The Parquet format specification describes files made of row groups, column chunks and pages, metadata written after the data and read first, nested data via Dremel-style record shredding, per-column compression and min/max statistics. Source
- The JSON Lines specification (source of jsonlines.org) requires UTF-8, one valid JSON value per line and a newline terminator, recommends a trailing newline for easy concatenation, the .jsonl extension and stream compressors such as gzip. Source
- Apache Arrow documentation says reading a subset of Parquet columns can be much faster than reading the whole file because of the columnar layout. Source
- Apache Arrow's Parquet dataset reader supports filtering on all columns using row group statistics. Source
- Hugging Face datasets can load local or remote csv, json, txt and parquet files with load_dataset, and describes Parquet as columnar and efficient for large datasets. Source
- Hugging Face datasets streaming lets you work with a dataset without downloading it. Source
- Hugging Face TRL documents a conversational dataset format in which each example holds a messages list of role and content turns. Source
- RFC 8259 notes that with IEEE 754 double-precision software, integers in the range from -(2^53)+1 to (2^53)-1 are interoperable in that implementations agree exactly on their values. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.