Data formats

Train/test split: how to split a dataset for AI training

Quick answer

A train/test split divides a dataset into a training portion the model learns from and a held-out test portion used only to measure how well it works on data it has never seen. Most projects also keep a third, smaller validation portion for tuning choices along the way.

By SourceX Editorial · Updated October 9, 2026

The three splits#

Training set: the largest share, used to fit the model.
Validation set: a smaller share used to compare settings and prompts during development.
Test set: held out until the end, used once for an honest quality measurement.

Why random row-by-row splitting can mislead#

If one conversation, customer or document spans many rows, a purely random split puts some of those rows in training and others in testing. The model has then effectively seen the test material, and scores look better than real-world performance. This is called leakage.

Group-aware splitting keeps every row that shares an ID — such as conversation_id — in the same split.
The overlap checker confirms afterwards that no record or group appears in two splits.

Choosing ratios#

Common starting points are 80/10/10 or 70/15/15 for train, validation and test. There is no universal right answer: smaller datasets often keep a larger test share so the measurement is meaningful, and very large ones can spare less.

Making splits reproducible#

A seeded split means the same file, ratios and seed always produce the same division, so a teammate or an auditor can recreate it exactly. The split tool below also writes a manifest listing which group went where.

A sensible order of operations#

Validate the file so every line parses.
Remove personal details, secrets and exact duplicates.
Split by group with a seed.
Run the overlap checker on the results before training.

Try it in your browser

Free tools. Your file is processed in this browser and not uploaded.

Train/test split tool (group-aware)Split data into train, validation and test with a seed, keeping each conversation or source in one split.Train/test overlap checkerFind exact records and shared group IDs that appear in both a training and a test sample.Dataset duplicate removerRemove exact duplicate records by whole record or chosen fields, with optional case and whitespace rules.

Frequently asked questions

What is a good train/test split ratio?

80/10/10 or 70/15/15 for train, validation and test are common starting points. The right choice depends on dataset size and how precise the final measurement needs to be.

What is data leakage?

Leakage is when information from the test set influences training — for example when rows from the same conversation land in both splits. It makes evaluation scores misleadingly high.

Why split by group instead of by row?

Rows that belong together (one conversation, one customer, one document) share context. Keeping them in one split prevents the model from being tested on material it partly saw during training.

Can I recreate the exact same split later?

Yes, if the split is seeded. The same input, ratios and seed always produce the same assignment, and the manifest records which group went to which split.

Sources

JSON Lines specification (jsonlines.org)

Tell us what your models need

Share scope, volume, language, format, timing and licensing requirements.

Request licensed data Talk to SourceX