How to test an AI estimating tool against your past jobs

Why test on past jobs instead of a vendor demo?#

Past jobs are the only test where you already know the right answer. A vendor demo uses jobs the vendor chose, priced with assumptions that may not match your labor rates, pricebook, suppliers or local code requirements. Your closed jobs carry all of that, plus the actual cost of doing the work.

A past-job test also measures the tool against two benchmarks instead of one. Comparing with your original estimate shows whether the tool drafts the way your estimators do. Comparing with actual job cost shows whether it would have priced the work more accurately, which is the question that decides margin.

Step 1: choose the test jobs#

Choose test jobs that look like the work you want the tool to handle, plus a few that should be hard for it. A set made only of simple jobs flatters every tool.

Exclude jobs with missing inputs. If the photos, measurements or plans your estimator used are gone, the tool cannot be judged fairly on that job, and a gap in the file will look like a tool error.

Step 2: rebuild bid-day inputs and hide the answers#

Bid-day inputs are whatever your estimator had before pricing: the customer's request, site photos, measurements, plans or takeoff files, equipment data plates and scope notes. Give the tool exactly those, and nothing created later.

Hiding the answers takes more care than it sounds. Final invoices, job cost reports, technician notes and change orders all reveal the outcome, so keep them in a separate folder the person running the test does not upload. Exports help here: ServiceTitan's guide to exporting Housecall Pro data says the jobs export includes invoice numbers, customers and line items and should be run in one-year batches if a report errors, and its FieldEdge guide says to set every Invoices filter to All so the export is not limited to the previous six months.

Do not hand the tool your field service login. ServiceTitan's terms of use bar customers from letting outside parties use its technology, and they name AI agents specifically. Read the AI vendor's data terms too: Procore's AI FAQ, for instance, notes that its own internal models may learn from what customers enter and receive in order to make Procore AI more accurate. Know where your test jobs go before you upload them.

Step 3: compare line items, quantities and totals#

Compare at several levels, because a correct total can hide offsetting mistakes. A tool that misses flashing but overprices labor can land near your number for the wrong reasons, and that luck will not repeat.

Step 4: log every miss and set decision rules#

A miss log turns impressions into a decision. Record each miss with the job, the item, the type of error and whether your estimator would have caught it on review.

Write the decision rules before you see results. For example: the tool may draft routine jobs if its misses are all ones an estimator catches on review; it is not used on larger jobs until missed-scope errors stop appearing; and it is retested on the same set after every major vendor update.

• 1Missed scope: an item the job needed that the tool left out.
• 2Unneeded scope: an item the tool added that the job did not need.
• 3Quantity error: the right item in the wrong amount.
• 4Price error: the right item and amount at the wrong unit price.
• 5Labor error: hours that do not match how your crews work.
• 6Unflagged uncertainty: a guess presented as fact.

Illustrative: a roofing contractor tests an AI takeoff tool#

Illustrative: a fictional residential roofing contractor considers an AI tool that reads aerial measurements and photos and drafts reroof estimates. Its estimator selects closed reroof jobs of different sizes and roof types, gathers the measurement reports and photos from each file, and moves invoices and job cost reports into a locked folder.

The tool comes close on squares and shingle quantities. It regularly misses decking replacement found during tear-off, extra flashing around chimneys and skylights, and permit and disposal costs that vary by town. Totals for simple roofs land near the original estimates; totals for complex roofs do not.

The contractor uses the tool for first drafts on simple reroofs only, adds a review checklist covering the items it misses, and keeps the test set to rerun when the vendor releases an update.

Your test set has value beyond the pilot#

A test set of closed jobs with bid-day inputs, the original estimate and the actual cost has the same structure AI developers look for when they license operational records: a real decision, the information behind it and the outcome. Keep it organized after the pilot; it is also your benchmark for the next tool you evaluate.

SourceX helps contractors license records like these without selling them outright, through the SourceX five-step transaction of Supply, Rights, Preparation, Approval and Delivery. The fit check uses metadata only, and Preparation removes customer names, addresses and photos that identify homes before anything is released.