Skip to content

Systems and records

Parquet or JSONL for a dataset? Decide by the buyer's workflow

By SourceX Editorial · Updated

10 sections

Short answer

Choose by the first job the buyer runs on the delivery. Use JSONL when each record is one self-contained task, such as a conversation, a ticket thread or an agent trajectory read whole by a training or evaluation loader. Use Parquet when the buyer will filter, sample, join or profile many fixed fields across millions of rows. Either way, ship string IDs, a manifest and a validator.

Value drivers

Dataset Parquet JSONL

This page covers parquet or JSONL for a dataset? Decide by the buyer's workflow.Illustrative

Key takeaways

  • Pick the format for the buyer's first job: whole-record reads favor JSONL, column scans, filters and joins favor Parquet.
  • Link tasks, steps and artifacts by stable string IDs, never by row order or file position.
  • Write IDs as strings, because JSON parsers agree on exact integers only up to about nine quadrillion.
  • Ship a package contract with schema and content versions, a manifest with hashes and a validator that needs no supplier systems.
  • If both formats are needed, make one canonical and generate the other with a checked, shipped script.

When should a business task dataset use Parquet versus JSONL?

Ask what the buyer does first with the files, not which format is better. If the first job reads each record whole, one task at a time, JSONL fits because one line is one task. If the first job scans a few columns across a large table to filter, sample, deduplicate or join, Parquet fits because it can read only the columns needed.

Business task data makes the choice harder than a flat table would. A support case, an invoice exception or a CRM workflow has a parent task, a variable number of steps and links to artifacts such as attachments, emails and system records. The format has to keep those links intact for the buyer's workflow while staying easy to validate.

The decision matrix below maps eight common buyer workflows to a first-choice format, the reason, what the package must include and the condition that flips the choice.

What does each format actually specify?

JSON Lines is a text convention with three rules: the file is UTF-8, each line is one valid JSON value, and lines end with a newline character. The jsonlines.org specification recommends a newline after the last value because it makes generating and concatenating files easier, suggests the .jsonl extension, and recommends stream compressors such as gzip for saving space. It says nothing about schema, types or required fields; those belong to your package contract.

Parquet is a binary columnar file format. Its specification describes a file as one or more row groups, each holding one column chunk per column, with column chunks divided into pages that are the unit of encoding and compression. File metadata is written after the data, and readers are expected to read that metadata first to find the column chunks they need. Parquet supports nested structures using the record shredding approach from the Dremel paper, compression chosen per column, and min and max statistics at several levels.

Those design facts drive the practical differences. Apache Arrow's documentation notes that reading a subset of columns can be much faster than reading a whole file because of the columnar layout, and Arrow's dataset reader can filter on any column using row group statistics. A JSONL reader has to parse every line to find anything.

Neither format defines meaning. Parquet enforces that a column holds integers or strings, but not that status_code 7 means closed, that a timestamp is UTC or that an amount is in cents. JSONL enforces even less: two lines can disagree on the type of the same field and both are valid JSON Lines. Units, time zones, enumerations and the relationship between records belong in the data dictionary and the validator, whichever container carries them.

Which format fits each buyer workflow?

Read across the row for the buyer's first job. The switch condition is the assumption that, if it turns out false, moves the recommendation to the other format.

Three rows cause most of the debate. Evaluation replay favors JSONL because an agent or workflow task is read as a unit: the harness needs the instruction, starting state, allowed tools, step history and expected end state together, and splitting them across tables means every replay begins with a join. Filtering and sampling favor Parquet because the selection reads a handful of fields, such as product area, outcome and date, across every task, and a columnar reader never touches the long text fields while doing it. Recurring appends are a draw: JSONL files can simply be extended, while Parquet deliveries add files per batch, which suits buyers whose lake already organizes data that way.

The matrix is a starting point, not a verdict. Two buyers with the same records can reasonably choose differently because their first jobs differ, and the agreement should record the choice and the reason so a later delivery does not change format by accident.

Which format fits each buyer workflow?
Buyer's first jobStart withWhySwitch if
Supervised fine-tuning on conversations or instructionsJSONLTrainers commonly expect one conversation per record, such as a messages list of role and content turnsThe corpus is very large and the buyer streams it from object storage; Parquet with a list of structs streams as well
Evaluation replay of agent or workflow tasksJSONL per taskThe harness loads one task with its steps, tools and expected end state as a unitHundreds of thousands of tasks need slicing by app, outcome or date before replay
Filtering, stratified sampling and deduplicationParquetOnly the filter and key columns are read, and row group statistics skip irrelevant dataThe buyer has no columnar tooling and the set is small enough to scan
Joining tasks to warehouse tables or other deliveriesParquetTyped columns and stable keys load directly into query enginesJoin keys live inside nested step records that the buyer will not flatten
Rights, privacy and quality review by peopleJSONL sampleA reviewer can open, search and comment on one record without codeReviewers already work in a notebook or SQL tool
Profiling distributions and drift across deliveriesParquetColumn statistics and fast column reads make per-field profiles cheapFields are mostly free text whose structure varies by record
Recurring appends from a live sourceJSONL or new Parquet files per batchJSONL files concatenate; a finished Parquet file is closed by its footer, so each batch arrives as new filesThe buyer's lake already partitions Parquet by date
Pretraining-style text at large scaleParquetCompression per column and column pruning keep text and metadata reads separateThe loader in the buyer's stack only reads line-delimited text

How do tasks stay connected to their artifacts in either format?

Give every task, step and artifact a stable string ID and link them by ID, never by row position or file order. Row order is not a contract in either format, and files get resharded, filtered and merged downstream.

Write IDs as strings even when the source system uses integers. RFC 8259, the JSON standard, notes that integers are interoperable across implementations only within the range from minus two to the 53rd power plus one to two to the 53rd power minus one; larger numeric IDs can be silently rounded by a JSON parser. Parquet can store 64-bit integers exactly, but a JSONL review sample of the same data would not, so strings keep both formats in agreement.

Keep large binary artifacts, such as PDFs, screenshots and audio, as separate files referenced by relative path and hash. Embedding them in a JSONL line as base64 inflates every read, and embedding them in a Parquet column makes column statistics meaningless for that field.

  • tasks: one row per task with task_id, workflow type, outcome, timestamps and source system.
  • steps: one row per step with step_id, task_id, order, actor role, action and resulting state, nested inside the task in JSONL or as its own table in Parquet.
  • artifacts: one row per file with artifact_id, task_id, relative path, media type, byte size and SHA-256.
  • Every reference resolves inside the package; nothing points to a supplier URL or internal share.

Can a buyer validate and rebuild the package without your internal systems?

Only if the package carries its own contract. The buyer should be able to run one validator against the delivered files and get a pass or a list of failures, without a call to the supplier and without access to the systems the data came from.

The checklist below applies to both formats; only the mechanism differs.

A good validator runs in a fixed order and stops on the first class of failure. It recomputes hashes and counts against the manifest, loads every file with the declared schema, checks uniqueness of IDs and resolution of every foreign key, verifies that each artifact path exists, and finally checks enumerations, units and time zones against the data dictionary. Shipping its report with the delivery means the buyer reruns the same checks and compares results instead of starting an investigation.

Can a buyer validate and rebuild the package without your internal systems?
Package elementIn a Parquet deliveryIn a JSONL deliveryValidator check
Schema versionArrow or Parquet schema file plus a version string in the manifestJSON Schema file plus a version string in the manifestEvery file declares the version it was written with
Content versionDelivery ID and date in the manifestDelivery ID and date in the manifestVersion matches the delivery note and the agreement schedule
Stable IDsString columns with uniqueness stated per tableString fields with uniqueness stated per record typeNo duplicates; every foreign key resolves
ManifestList of files with row counts and SHA-256List of files with line counts and SHA-256Counts and hashes recomputed and matched
Relative pathsArtifact paths relative to the package rootSameEvery path exists inside the package
Field semanticsData dictionary: meaning, unit, time zone, allowed valuesSame dictionaryEnumerations and units checked against the dictionary
RolesActor role column with a closed listRole field on each turn or stepNo role outside the list; no personal names in role fields
Runnable validatorScript that reads only the delivered filesSame scriptExit code and report shipped with the delivery

Which format wins when one support history serves two buyer jobs? An illustrative scenario

This scenario is invented to show the decision. A software company licenses de-identified support history: cases, the message turns inside them, internal notes, status changes and attachments. The buyer has two jobs: fine-tune a support assistant on resolved conversations, and first build an evaluation slice stratified by product area, severity and outcome.

The first job the buyer runs is the stratified selection, which reads 4 columns across all cases and joins to a product table. That favors Parquet as the canonical delivery: a cases table, a turns table and an artifacts table, linked by string IDs. The fine-tuning job then needs one conversation per record in a messages format, which the buyer, or the supplier as a derived file, produces from the turns table.

The deciding check is whether the derived JSONL can be rebuilt exactly from the canonical tables. The validator regenerates it, compares case counts and per-case turn counts, and hashes the output. In this illustration, 100 percent of cases in the derived file must match the canonical tables; any mismatch blocks the delivery rather than being noted.

The exception: if the buyer had only wanted fine-tuning on a few thousand conversations, the canonical Parquet tables would add work without benefit. A single JSONL file with a published schema and a manifest would be the better package.

Either way, the choice is recorded in the agreement schedule with the reason, the canonical format named, and the derived file listed as generated, so a later delivery cannot quietly switch format or let the two drift apart.

Which workflow assumption would flip the format choice?

Write the assumptions into the delivery note so the buyer can see which one their workflow violates. Most format disputes come from an assumption nobody stated, not from the format itself.

Steps

Which workflow assumption would flip the format choice

  • Size: below the point where a full scan is cheap for the buyer, Parquet's column pruning saves little and JSONL's readability wins.
  • Access pattern: whole-record reads favor JSONL; column scans, filters and joins favor Parquet.
  • Nesting: deep, irregular structures are natural in JSON; in Parquet they need an explicit nested schema that every writer follows.
  • Schema stability: a field set that changes every delivery is easier to version in JSONL with JSON Schema; a stable field set benefits from Parquet's typed columns.
  • Tooling: Hugging Face datasets loads both JSON and Parquet files and can stream a dataset without downloading it first, so many training stacks accept either; confirm the buyer's actual loader before deciding.
  • Review: if people outside engineering must read records, ship a JSONL sample whatever the canonical format is.

How large does a dataset need to be before Parquet pays off?

There is no universal row count, so measure it in the buyer's own stack rather than trusting a rule of thumb. The gain from Parquet comes from skipping columns and row groups the job does not need, which depends on how many fields the job reads, how wide the text fields are and which engine does the reading.

A fair test takes a representative sample of the actual delivery, writes it in both formats with the compression the delivery would use, and times the buyer's real first job on each: the selection query, the loader or the profiling script. Record the engine and version, the sample size and the hardware, and repeat each run several times so one slow read does not decide the question.

If the difference on the sample is small, prefer the format that is easier to validate and review. If it is large and grows with sample size, the full delivery will show the same pattern, and Parquet earns its extra tooling.

When is shipping both formats worth the extra work?

Ship both only when the buyer's workflow genuinely has two first jobs, and then make one canonical and the other derived. Two independently produced files drift: a fix applied to one is forgotten in the other, and the buyer cannot tell which is right.

The rule is simple. Name the canonical format in the agreement schedule, generate the derived files with a script included in the package, and make equivalence a validator check with counts and hashes. If the derived format is only for human review, cap it as a sample and say so in the manifest.

Asking for licensed business task data in the format your workflow needs

Write the request around the work, not the file extension. A sentence such as first we select an evaluation slice by product area and outcome, then we fine-tune on resolved conversations tells a supplier more than a bare request for Parquet, and it lets the supplier flag early when the source system cannot produce a field the workflow depends on.

Put the matrix row that matches your first job into the data request, along with the package elements you expect to validate. Format is easy to change before preparation starts and expensive to change after a delivery has been reviewed and signed off.

Because SourceX looks for each dataset at U.S. businesses only after a buyer asks, the request form, which covers scope, volume, language, format, timing and licensing needs, is the place to state the canonical format and package contract from this page.

Frequently asked questions

Is JSONL or Parquet better for fine-tuning data?

For most fine-tuning on conversations, JSONL is the simpler fit because each line holds one complete example, such as a messages list of role and content turns. Parquet also works and is often used for very large corpora. Check what the trainer's loader expects before deciding.

Can Parquet store nested conversations and agent steps?

Yes. Parquet was designed for nested data and uses the record shredding approach described in the Dremel paper. A conversation can be a list of structs in one column, though every writer has to follow the same nested schema.

Why should IDs be strings in a JSONL delivery?

Because JSON parsers commonly use double-precision numbers. RFC 8259 notes that only integers within plus or minus two to the 53rd power, less one, are read identically by all implementations, so larger numeric IDs can be rounded without warning.

Should a dataset be delivered in both Parquet and JSONL?

Only when the buyer has two distinct first jobs. Make one format canonical, generate the other with a script shipped in the package, and check equivalence with counts and hashes so the two cannot drift.

Can a JSONL file be appended to, and a Parquet file?

A JSONL file can be extended by adding lines, and files concatenate when each line ends with a newline. A Parquet file ends with its metadata footer, so new data normally arrives as new Parquet files that the reader treats as one dataset.

Sources

  • The Parquet format specification describes files made of row groups, column chunks and pages, metadata written after the data and read first, nested data via Dremel-style record shredding, per-column compression and min/max statistics. Source
  • The JSON Lines specification (source of jsonlines.org) requires UTF-8, one valid JSON value per line and a newline terminator, recommends a trailing newline for easy concatenation, the .jsonl extension and stream compressors such as gzip. Source
  • Apache Arrow documentation says reading a subset of Parquet columns can be much faster than reading the whole file because of the columnar layout. Source
  • Apache Arrow's Parquet dataset reader supports filtering on all columns using row group statistics. Source
  • Hugging Face datasets can load local or remote csv, json, txt and parquet files with load_dataset, and describes Parquet as columnar and efficient for large datasets. Source
  • Hugging Face datasets streaming lets you work with a dataset without downloading it. Source
  • Hugging Face TRL documents a conversational dataset format in which each example holds a messages list of role and content turns. Source
  • RFC 8259 notes that with IEEE 754 double-precision software, integers in the range from -(2^53)+1 to (2^53)-1 are interoperable in that implementations agree exactly on their values. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies