How Much Data Do You Need to Fine-Tune a Vision-Language Model?

Published reference points: hundreds to hundreds of thousands

Public results span three orders of magnitude, and the spread is driven by task breadth and data quality, not by model family alone. LLaVA trained its projector on 595K image-text pairs, then fine-tuned the projector and language model on 158K generated instruction-following samples split into conversations, detailed descriptions and complex reasoning [2]. InstructionGPT-4 (later titled MM-LIMA) fine-tuned MiniGPT-4 on only 200 quality-selected examples, about 6% of the original alignment instruction data, and reported better VQA and GPT-4 preference results than the full set [1].

The text-only literature points the same way. LIMA fine-tuned a 65B model on 1,000 curated prompt-response pairs [3], and a 2024 question-answering study found most models peaked or nearly peaked at about 60 training samples for that narrow task [5]. Read these as evidence that curation dominates headline counts, not as targets: each result came with a strong base model, a narrow scope or heavy selection, and evaluation on benchmarks that may not resemble your images. For the text-only version of this question, see how much data to fine-tune an LLM.

Size by coverage cells, not a headline image count

The defensible unit for a domain VLM purchase is the coverage cell: one task crossed with one visual condition crossed with one source context. A field-service model that reads equipment nameplates, flags corrosion and answers technician questions has at least three tasks. Each of those tasks then meets different lighting, camera types, occlusion, glare, resolution and site layouts, and each source company photographs differently.

A headline count hides missing cells. Ten thousand images from one well-lit site can underperform two thousand images spread across eight sites, because the model has never seen a night photo, a phone with a cracked lens or a nameplate in a second language. Write the request as a matrix, and attach a minimum per cell rather than a total. The visual instruction tuning data guide covers what goes inside each example; this page covers how many.

What changes the per-cell minimum

• Output form. Short-answer VQA and classification-style prompts need fewer examples per cell than free-form descriptions, multi-turn dialogue or grounded answers with bounding boxes.
• Distance from pretraining. Natural photos of common objects sit close to web pretraining data. Microscopy, thermal imagery, CAD renders, engineering drawings, dense forms and proprietary UI screenshots sit far away and need more.
• Fine-grained distinctions. Telling part number A from a near-identical part B, or a hairline crack from a scratch, needs many hard negatives, not just more positives.
• Text in the image. OCR-heavy tasks (invoices, labels, gauges, screens) are sensitive to resolution and the image preprocessing of the target model, so sample at the resolution you will deploy.

LoRA versus full fine-tuning changes what volume buys you

Adapter methods such as LoRA and QLoRA update a small set of parameters, so they are usually tried first on a few thousand examples, while full fine-tuning of the language model and vision tower is generally reserved for larger, more varied sets. Treat that as a working hypothesis for your stack, not a rule: the right answer depends on the base model, which modules you unfreeze (projector only, language model, vision encoder) and how far your images sit from its pretraining distribution.

The practical consequence for buyers is sequencing. Run the adapter experiment first on a pilot slice; if the learning curve is still rising steeply at the full pilot size, or if the errors concentrate in visual perception rather than answer formatting, you have evidence that you need more data, more cells, or an unfrozen vision encoder. Full fine-tuning on a small set also raises the risk of catastrophic forgetting of general skills, so hold out a general benchmark alongside your domain eval. For definitions, see the SourceX fine-tuning glossary entry.

Pilot first, then fit a learning curve

A paid pilot plus a learning curve is the cheapest way to turn sizing from a guess into an estimate. Generalization error tends to follow a power law in training-set size across domains, including image tasks [4], so a few points on the curve let you extrapolate how much additional data buys a given improvement.

1. Buy or assemble a pilot that covers every coverage cell thinly, even at a handful of examples per cell.
2. Train at 10%, 25%, 50% and 100% of the pilot, with the same seed set and hyperparameters, and score each run on the same held-out domain eval.
3. Plot error against examples on log-log axes per task. Flat curves mean more of the same data will not help; switch to new cells, harder negatives or better labels.
4. Extrapolate to your target error and convert that into examples per cell for the full request.

Quality multiplies volume. Near-duplicate images (burst photos, the same asset photographed weekly, screenshots of identical screens) inflate counts without adding information, and deduplication improves text models while reducing memorization [6]. Run perceptual-hash or embedding-similarity dedup before you count anything, and ask suppliers to report counts after dedup.

Budget evaluation data separately from training data

Evaluation data is a separate line item, and it should come from sites, time periods or source companies that are not in the training set. Public multimodal benchmarks may already be in a base model's training data; researchers have documented contamination of multimodal benchmarks in VLM training data [7]. A model that scores well on a public VQA set can still fail on your nameplates.

Plan for a held-out set that covers every coverage cell, with answers written or verified by domain experts, and with a slice held back from every vendor and annotator who touched training data. A few hundred carefully adjudicated items per task is a common starting point for a decision-grade eval; scale it to the confidence interval you need for a go or no-go call. Sourcing options are covered in private evaluation sets for multimodal models.

A sizing worksheet for an image-text request

Fill in the matrix below before you contact suppliers; it doubles as the core of the request. The structure, not the numbers, is the point.

Illustrative example: invented to show structure; it does not describe an available dataset.

Totals fall out of the matrix: cells times per-cell targets, plus the eval column, after dedup. Add fields that let you audit coverage later: capture device, timestamp, site identifier (pseudonymized), resolution, annotator identifier and review status. Managed tuning services such as Vertex AI expect JSONL records that reference image files by URI alongside the prompt and response, so confirm the target format early [8]. The multimodal data request specification template expands this into a full request, and the completed matrix is what to bring when you submit a data request to SourceX.

Failure modes that make the count wrong

Most sizing mistakes are coverage or leakage problems that show up only after training. Watch for these before you scale a purchase:

• Site leakage. Random splits put photos of the same asset in train and eval, so scores overstate generalization. Split by site, asset or time.
• Synthetic answers without verification. Instruction data generated by a text-only model from captions, as in LLaVA's pipeline [2], scales cheaply but inherits caption errors; domain tasks need expert-verified answers on at least the eval set.
• Class imbalance hidden inside cells. Ninety-five percent "no defect" images teach the model to say no.
• Answer style drift. Mixed annotator conventions (units, JSON keys, hedging language) cost more than missing examples. Lock a style guide before labeling.
• Unremoved personal details. Faces, badges, screens and EXIF GPS in field photos need handling before delivery; see de-identifying multimodal records.

When translating a curve into a budget, the broader guide to sizing a data purchase covers phasing and option-to-expand structures. If you are building a detector or classifier rather than an instruction-following model, use how many images to train a computer vision model instead.

Sourcing image-text data for a VLM fine-tune

SourceX sources operational datasets from US companies on request, including documents, engineering records and new recordings of hands-on work, and manages licensing from assessment through ongoing purchases. Each dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, with personal details removed or replaced before delivery; a request does not guarantee a match. Bring your coverage matrix and describe the image-text data your VLM fine-tune needs. More context sits in the multimodal data hub.