Getting started
Utility dataset pilot evidence: what an energy data pilot should prove
By SourceX Editorial · Reviewed by Noah Loul ·
10 sections
Short answer
An energy workflow data pilot should prove, on a small bounded slice, that cases link asset, reading, tariff, contract and work order evidence; that every fact is dated to the event; that estimated and corrected readings keep their quality flags; that scope stays administrative; and that infrastructure, customer and field-safety boundaries held. Model usefulness is the buyer's own pre-registered test on held-out cases. This guide gives the method, not results.
Process
Utility dataset pilot
Key takeaways
- Prove completeness, dating, reading quality, bounded scope and boundaries; leave model gains to the buyer's pre-registered test.
- Join readings, rates and contract terms as of the event date, and keep later corrections as separate dated records.
- Keep reading quality codes; dropping them turns estimates into apparent facts.
- Exclude infrastructure detail, fine-grained customer usage and any field or switching procedure before selecting records.
- Write the protocol, holdout and reporting terms down before any record is chosen.
- Stop and narrow scope when a scan finds protected content; do not rely on case-by-case cleaning.
What should an energy workflow data pilot demonstrate?
An energy workflow data pilot should prove five things on a small, bounded slice of records: that cases link asset, account, reading, tariff, contract and work order evidence; that every fact is dated so the rule in force on the event date can be recovered; that corrected and estimated readings keep their quality flags; that the task scope is narrow and administrative; and that confidentiality and safety boundaries held throughout. Whether the records improve a model is the buyer's own test, run on a held-out set agreed in advance.
This guide is for utilities, energy retailers, energy service companies and large energy users weighing a pilot before licensing operational records. It covers administrative and back-office workflows such as billing exceptions, meter data corrections, tariff application, service orders and contract administration. It does not cover switching, clearances or any field procedure, which stay out of scope for reasons explained below.
No pilot results are reported here. The plan describes what to prove and how to show it, so the evidence can be reviewed before anyone draws conclusions.
Which records make an energy administrative task checkable?
A task is checkable when a reviewer can recompute or confirm the right answer from the case itself. In energy administration that almost always means joining several systems: the customer information and billing system, the meter data management system, the asset register or GIS, the work management system and the contract repository.
Identifiers are the first thing to test. Account, premise, service point and meter are different entities: an account can move between premises, a premise can have several service points, and a meter can be exchanged while the service point stays the same. A pilot should show that each case uses the right key for each join, and that history survives a move-in, move-out or meter exchange instead of being overwritten by the current mapping.
Meter readings are where most pilots go wrong, because the same interval can be read, estimated, corrected and re-billed. The Green Button ESPI schema published by the Green Button Alliance shows how much a reading can carry: its quality-of-reading codes distinguish valid data, manually edited values, values estimated from a reference day or by linear interpolation, questionable data that failed checks, raw data that has not been validated, validated data, and data that failed a check but was verified as actual usage. A pilot that drops those flags turns estimates into apparent facts.
| Evidence link | Typical system | What it lets a reviewer check |
|---|---|---|
| Asset or meter identifier with install and removal dates | Asset register, meter data management | Which device produced a reading on a given date |
| Reading with revision history and quality code | Meter data management | Whether a bill used an estimate, a correction or an actual read |
| Tariff or rate with effective and end dates | Billing rate tables, published tariff | Which rate applied to each day of a billing period |
| Contract term or special agreement | Contract repository, CRM | Whether a negotiated term overrides the standard rate |
| Work order with status timestamps | Work management system | Whether a meter exchange or field visit explains a reading gap |
| Approval and exception record | Billing exception queue, workflow tool | Who authorized a rebill, write-off or adjustment and why |
| Outcome | Billing ledger, customer account history | The final corrected bill and any customer contact |
Why do corrected readings and tariff dates decide the label?
The correct answer to most billing and meter tasks depends on what was true on a date, not what is true now. If a rate changed mid-period, a bill must be prorated across both; if a reading was estimated and later replaced by an actual read, the rebill compares the two. A pilot that joins current tariffs and latest readings to historical bills will mark correct work as wrong.
Public rate data shows the pattern. The OpenEI Utility Rate Database API, which publishes U.S. utility rate structures, lets a caller ask for rates effective on a given date, a reminder that a rate is a dated record with a start and, once superseded, an end. Internal rate tables need the same treatment, and so do contract terms, which can start, renew or lapse inside a billing period.
So the pilot should demonstrate as-of joins: every reading, rate and contract term in a case is the version that applied on the event date, with the later corrections kept as separate, dated records. Reviewers can test this directly by recomputing a sample of bills from the case contents.
Where do the safety and confidentiality boundaries sit?
Three boundaries decide what a pilot may contain, and each should be written into the plan before any record is selected.
Infrastructure information comes first. FERC's current CEII regulation, 18 CFR 388.113, covers specific engineering, vulnerability or detailed design information about critical infrastructure that could be useful to someone planning an attack, beyond the general location of a facility. NERC's CIP-011 standard requires responsible entities to identify and protect BES Cyber System Information. Work orders, asset records and outage notes can carry exactly this kind of detail: one-line diagrams, protection settings, network addresses or access procedures. The pilot plan should exclude those fields and the documents that contain them, and the utility's CIP or security lead should confirm the exclusion list.
Customer energy usage is the second boundary. Green Button's Connect My Data model has a retail customer authorize a third party to receive usage information from the utility, which reflects how sensitive interval data is treated. Usage at fine intervals can reveal occupancy and routines, so the plan should state how account and premise identifiers are removed or tokenized and whether interval data is aggregated.
Field safety is the third. OSHA's 1910.269 standard covers the operation and maintenance of electric power generation, transmission and distribution lines and equipment. A dataset of administrative work should not train a model to plan or instruct that work. Keep switching orders, clearance and lockout records, and crew safety procedures out of the pilot, and say so in the scope.
What should the domain reviewed utility and risk acceptance plan include?
The plan below states each claim the pilot must support, the evidence that supports it, the test a reviewer runs, who reviews it and what counts as failure. Agree it with the prospective buyer before records are selected, so the result cannot be argued after the fact.
| Pilot claim | Evidence to produce | Acceptance test | Reviewer | Fails if |
|---|---|---|---|---|
| Cases are complete | Linked records per case across the seven evidence links | Trace a random subset from request to outcome | Data lead and billing operations | Any traced case is missing an outcome or a governing rule |
| Facts are dated | Effective and end dates on rates, contracts and readings | Recompute a sample of bills from case contents | Billing analyst | Recomputed amounts differ without a recorded reason |
| Reading quality is preserved | Quality codes and revision history on every reading | Compare code distribution with the source system | Meter data lead | Estimated or edited reads appear unflagged |
| Scope is bounded | Written list of task families, territories, periods and fields | Check every record against the scope list | Operations owner | Any record falls outside the list |
| Infrastructure information is excluded | Exclusion list and scan results for diagrams, settings and addresses | Targeted search of notes and attachments | CIP or security lead | Any protected detail appears |
| Customer data is protected | De-identification method and re-identification test | Attempt linkage on a sample with known identifiers | Privacy lead | A premise or customer can be re-linked |
| Safety boundary holds | List of excluded field procedures and record types | Search for switching, clearance and lockout content | Safety lead | Field procedure content is present |
| Rights are confirmed | Contracts, regulatory orders and third-party terms covering the records | Counsel review against the scope list | Counsel | A source is restricted or unclear |
| Usefulness is measured by the buyer | Buyer's pre-registered test, metric and held-out set | Buyer runs its test and reports against the plan | Buyer's technical lead | The test or metric changes after records are seen |
How should the pilot be sized and run without overreaching?
Keep the pilot narrow enough that every case can be reviewed. One task family, one service territory or business unit, and one continuous period is a sensible shape; the period should cover at least one rate change or seasonal billing pattern so the dated-facts claim is actually tested.
At a glance
How should the pilot be sized and run without overreaching
Write the protocol down before selecting records: the question the pilot answers, the population it draws from, the selection rule, the variables carried in each case, the comparison the buyer will run, the metrics, how uncertainty will be reported and the known limits. The NIST AI Risk Management Framework's MEASURE function calls for documenting test sets, metrics and tool details used in testing, and for documenting limits on how far results generalize beyond the conditions tested. A pilot plan that does the same is easier to accept and harder to over-read.
Hold out a set of cases the buyer never sees until the test. Agree how the buyer will report results back, if at all, and what each party may do with the pilot records afterward; those terms belong in the pilot agreement.
- Question: the single task the pilot records are meant to support, stated in one sentence.
- Population and selection: the territory, account class, period and the rule used to draw cases, including how rare cases are handled.
- Variables: the fields carried per case, with their source system and as-of date logic.
- Comparison: what the buyer will compare against, such as its current data or a public dataset, on the same held-out cases.
- Metrics and uncertainty: the metric, how it is computed and how its uncertainty will be reported, for example an interval rather than a single figure.
- Limits: what the pilot cannot show, such as performance in other territories, account classes or periods.
- Change control: what happens if the plan has to change after records are seen, and who must agree.
Which dated rule, asset fact or safety boundary changes the permissible next step?
Each case should let a reviewer say what the next step was allowed to be, and why. These rules flag the facts that most often change the answer.
Checklist · 6 items
- A rate or rider change inside the billing period changes the correct bill; the case must hold both versions and their dates.
- A meter exchange changes which device's readings apply; the case needs the work order and the install and removal dates.
- An estimated reading later replaced by an actual read changes the rebill; keep both values with their quality codes.
- A negotiated contract term can override the published rate; the case must show the term and its effective dates.
- An approval threshold changes who may authorize an adjustment; keep the approver's role and the threshold in force.
- Any step that would require field work, switching or a safety procedure is outside the dataset's task scope; the correct next step in the data is a hand-off to the responsible team, not an instruction.
0 of 6 ticked. Tick each item as you go.
Worked plan (illustrative): estimated-read rebills for commercial electric accounts
This plan is illustrative and uses counts and durations only. A utility proposes a pilot on rebills caused by estimated readings for small commercial electric accounts in one territory over 18 months, a window that includes one general rate change and one meter exchange program.
Each case would link the account (tokenized), the meter and its install dates, the estimated and actual readings with their quality codes, the rate versions in force, any special contract term, the exchange work order where relevant, the rebill approval and the final bill. The plan excludes outage records, protection or network details, switching and clearance records and all interval data below daily totals.
Acceptance would be run on a random draw of a few hundred cases, illustrative only, with billing analysts recomputing a subset and the security lead scanning notes. The buyer pre-registers a single task and metric, for example whether a model proposes the correct rebill basis, and evaluates on a held-out set. No result is reported in this guide.
The exception: accounts served under a negotiated contract with confidentiality terms that cover usage and pricing. Those cases are dropped from the pilot unless counsel confirms the contract allows the use, even though they are some of the most interesting rebills.
When should a pilot stop, narrow or bring in specialist review?
Stop and fix the source when protected infrastructure detail, customer identifiers or field procedure content turns up in a scan; narrow the scope rather than relying on case-by-case cleaning. Pause for counsel when records come from regulatory proceedings, joint ownership arrangements, wholesale or interconnection agreements, or contracts with confidentiality terms. Bring in a domain reviewer whenever recomputed bills disagree with recorded ones for reasons nobody can explain.
Narrowing is usually the better outcome than abandoning the pilot. If interval data proves too sensitive, move to billing-period totals; if one territory's work orders carry too much infrastructure detail, choose another; if a task family keeps pulling in field procedures, drop it and keep the purely administrative ones. Record each narrowing decision and its reason in the plan, because the same reasons will shape the full dataset.
Recheck the plan when tariffs, asset records, readings practices, contracts, operating conditions or the scope change. A pilot plan is tied to the systems and rules of its period; it is evidence about that slice, not a statement about every record the company holds.
From pilot evidence to a SourceX assessment
Energy companies that move from a pilot to a license can work through SourceX, which runs the sourcing and qualification, rights review, contracting, data preparation, delivery and payment for selected operational records licensed to AI developers. Eligibility is checked by assessing the company first; no records are uploaded for that step.
The acceptance plan above is useful input to that conversation because it states scope, exclusions and evidence in advance. Delivery waits for the company's approval of scope and terms and its signature, and archives too large to send over a network remain in the company's storage or travel on encrypted drives, because SourceX does not host them.
Frequently asked questions
What is utility dataset pilot evidence?
It is the set of records and checks that show a small slice of energy workflow data is complete, correctly dated, quality-flagged, tightly scoped and free of protected infrastructure, customer and safety-procedure content. It supports a decision to scope a larger dataset; it is not a claim that a model improved.
Should an energy data pilot report model accuracy?
Only if the buyer runs a test agreed before the records were seen, on a held-out set, and the parties agreed to share the result. The supplier's pilot evidence should focus on completeness, dating, quality flags and boundaries, which it can show directly.
Can interval meter data be part of a pilot?
Sometimes, but it needs care. Fine-grained usage can reveal occupancy and routines, so many pilots use daily or billing-period totals with tokenized accounts. Counsel and the privacy lead should decide based on the applicable rules and customer terms.
How do I keep critical infrastructure information out of a dataset?
Write an exclusion list before selecting records: diagrams, protection settings, network addresses, access procedures and similar detail, plus the documents that carry them. Have the CIP or security lead confirm it, then scan notes and attachments and narrow the scope if anything appears.
Why do reading quality codes matter for AI training data?
Because an estimate and an actual read look identical once the code is dropped. Standards such as Green Button ESPI distinguish estimated, edited, raw, validated and verified readings, and a model learning billing corrections needs those distinctions to learn the right rule.
Sources
- The Green Button ESPI schema defines quality-of-reading codes including valid, manually edited, estimated using reference day, estimated using linear interpolation, questionable, raw, validated and verified. Source
- Green Button Connect My Data lets a retail customer authorize a third party to access energy usage information directly from the data custodian. Source
- The OpenEI Utility Rate Database API lets callers request rates effective on a specified date. Source
- FERC's CEII definition covers specific engineering, vulnerability or detailed design information about critical infrastructure that could be useful to a person planning an attack and that gives more than the general location of the infrastructure. Source
- NERC CIP-011 aims to prevent unauthorized access to BES Cyber System Information by requiring entities to identify and protect it. Source
- OSHA 1910.269 covers the operation and maintenance of electric power generation, control, transformation, transmission and distribution lines and equipment. Source
- The NIST AI RMF MEASURE function calls for documenting test sets, metrics and tool details used in testing and the limits of generalizing beyond tested conditions. Source
See if your company qualifies
A short company assessment. No data uploads are needed.