Systems and records
Creative version selection benchmark: can an agent find the approved version without the filename?
By SourceX Editorial · Updated
10 sections
Short answer
To test whether an agent identifies the approved version without relying on filenames, give it a pool of candidate files for one creative job, strip or scramble the filenames, and ask which file was approved for a stated use on a stated date. Score three things separately: the file it picks, the review or rights record it cites as proof, and whether it declines when no candidate is cleared for that use.
Evaluation split
Creative version selection
Key takeaways
- Treat version selection as a judgment across a pool of files, scored against review decisions, not as a filename lookup.
- The proofing or review decision is the authority for approval; embedded metadata links versions but almost never records approval.
- Approval is scoped: the gold label should say approved for which channel, territory and date window.
- Decoys such as renamed copies, renditions, localized variants and expired rights decide whether the benchmark measures anything.
- Score the pick, the cited evidence and abstention separately, and report picks of a rejected or out-of-window version as their own error.
How do you test whether an agent identifies approved versions without relying on filenames?
Build each test item as a choice among several real files from one creative job, hide or scramble the filenames, and ask a question with a use and a date attached: which of these is the version approved for paid social in the United States on a given day. The gold answer names one file, the review decision that approved it, and any rights record that limits where it can run. An agent passes only when it picks that file and cites that evidence.
Filenames are the shortcut you are trying to rule out. Creative teams rename exports by hand, so a folder can hold spot_final, spot_final_v2 and spot_FINAL_approved, and none of the three names says which cut the client signed off. An agent that sorts by the word final or by modified date will look accurate on a tidy folder and fail on a real one. Removing the filename forces the agent to read the evidence a creative operations lead would check.
The rest of this guide sets out where approval evidence lives in a creative workflow, what embedded metadata can and cannot prove, how to scope approval by channel and rights, a field-level item specification, a decoy catalog, a scoring rule, an illustrative item and the cases to leave out.
Which record decides that a creative version is approved?
The review or proofing decision decides it, and every other signal ranks below that. In a proofing tool, approval is an explicit decision by a named reviewer on a specific version of a file. Adobe Workfront's standard proof status options, for example, are Pending, Approved, Approved with Changes, Changes Required and Not Relevant, and Adobe documents that the overall status of a proof is driven by the worst-case participant, so one Changes Required outweighs several approvals. Account administrators can rename or hide those decision labels, so the benchmark should store the label as the tool showed it and map it to a fixed set of states.
Rank the evidence before labeling and give labelers the ladder below. When two sources disagree, the higher rung wins and the item records the conflict, because conflicts are where agents fail.
| Rung | Evidence source | What it can establish | Typical gap |
|---|---|---|---|
| 1 | Proofing or review decision on a specific version | Who approved which version, when, and with what conditions | Approval given by email or in a meeting and never entered |
| 2 | Rights and clearance record (talent, music, stock, territory, window) | Where and until when an approved version may run | Held in contracts or spreadsheets with no link to the file |
| 3 | Delivery or trafficking record | Which file actually shipped to which channel or outlet | Shows what ran, which may differ from what was approved |
| 4 | Asset management status field | Workflow state such as in review, approved, archived | Set by hand, often lagging the review decision |
| 5 | Embedded metadata and provenance manifests | Lineage between versions and renditions | Rarely records approval at all |
| 6 | Filename, folder and modified date | Nothing reliable about approval | Renamed, copied and re-exported at will |
What can embedded metadata prove about a version, and what can it not?
Embedded metadata can prove lineage, meaning which file came from which, but it cannot prove approval. That makes it valuable for building candidate pools and dangerous as a label. In Adobe's XMP Media Management namespace, xmpMM:DocumentID is the common identifier for all versions and renditions of a resource, xmpMM:InstanceID changes each time the file is saved, and different renditions are expected to carry different DocumentIDs. xmpMM:VersionID holds a version identifier and xmpMM:Versions holds a version history, but the specification warns that the history may be incomplete.
The XMP history records actions, not decisions. Its ResourceEvent structure defines actions such as converted, copied, created, edited, published and saved, each with a timestamp. A published event says a file was published, not that a client approved it. Rights fields are similarly limited: xmpRights:Marked is meant to be omitted when the rights state is unknown, and Adobe states the rights properties are a means of rights expression, not access control.
Content Credentials work the same way at a deeper level. In the C2PA 2.2 specification, each ingredient a manifest references must have a relationship of parentOf, inputTo or componentOf, and a manifest may have at most one parentOf ingredient. That gives an agent a verifiable chain from an export back to its parent, which is useful for deciding whether two files are the same version. None of those fields says a reviewer approved anything, so record them as lineage evidence and keep approval on rung 1.
Approved for which channel, territory and window?
Approval in creative work is almost always conditional, so the gold label must name the use it covers. A cut can be approved for broadcast but not for social, cleared in one country and not another, or approved while a music license runs and unusable after it ends. A benchmark that stores a single approved flag will mark an agent wrong for refusing an out-of-window file and right for picking a file the business could not lawfully run.
Use identifiers that separate versions from formats. The entertainment identifier registry EIDR distinguishes an Edit, a version of a work such as a director's cut, from a Manifestation, the playable digital form; its data-model reference treats creative versions and technical encodings as separate records and describes relationships used by distribution and rights systems. Apply the same split: a localized subtitle file or a different encoding of the same cut is a manifestation of one approved edit, while a recut is a new version that needs its own decision.
Carry rights fields into the item even when they are empty. The IPTC Photo Metadata Standard includes a Model Release Status property and, since version 2023.1, a Data Mining property whose value says whether data mining is prohibited or permitted; IPTC notes that leaving it empty does not grant permission. Treat an empty rights field as unknown, never as cleared, and let the rights reviewer, not the agent or the labeler, settle what a contract allows.
The version authority and evidence benchmark, field by field
Each benchmark item is one record with the fields below. Build pools from whole jobs, not random pairs: pull every file attached to the job in the review tool, the asset library and the delivery log, match copies by content hash and lineage IDs, then choose candidates. Split train and test by client or campaign so the same job never appears on both sides. Version the benchmark with a date and keep old versions when a rights record later changes a gold answer.
Most proofing and work management tools can export a job's version list with decisions and timestamps, and most asset libraries can export lineage and status fields. Join those exports on the file, not the name: a content hash or the embedded identifiers tie a proofing upload to the same file in the library and the delivery log, even when each system renamed it.
| Field | What it holds | Validation check | Why it matters |
|---|---|---|---|
| item_id, question_date, intended_use | Stable ID, the date asked about, and channel plus territory | All three present; use drawn from a closed list | The same pool has different answers for different uses and dates |
| candidates | Two to eight file IDs with scrambled names, format, duration or dimensions, content hash | Every ID resolves; original filenames held back in a sealed column | Removes the filename shortcut while keeping it auditable |
| gold_selection | One candidate ID or none_cleared | Exactly one value | Tests abstention as well as choice |
| version_state per candidate | Draft, changes required, approved with changes, approved, superseded, rendition | One state per candidate; mapped from tool labels | Lets errors be diagnosed, not just counted |
| review_decisions | Decision label as shown, reviewer role, version number, timestamp | At least one decision for the gold file unless none_cleared | Rung 1 evidence the agent must cite |
| rights_constraints | Territory, channel, start and end date, release status per element | Unknown allowed, blank not allowed | Separates approved from usable |
| lineage | DocumentID, InstanceID, parent references, provenance manifest status | IDs present where the file carries them | Shows which files are the same version in another format |
| delivery_records | Destination, file ID shipped, ship date | Each record resolves to a candidate or is flagged external | Catches files that ran without approval |
| decoy_tags | Decoy family for each wrong candidate | Values from the decoy catalog | Reports accuracy by difficulty |
| label_provenance | Two labelers, adjudicator, rights reviewer sign-off where rights decide | Disagreements resolved and recorded | Makes the gold answer defensible |
Which decoys expose an agent that is still guessing?
Decoys should be files a busy person could plausibly pick, and the catalog should be sampled in proportions declared before any agent runs. The most useful decoys are the ones where the filename or the newest timestamp points the wrong way.
| Decoy family | What the wrong file looks like | Evidence that rules it out |
|---|---|---|
| Named final, not approved | Original name contained final or approved; review decision was Changes Required | Review decision on that version |
| Approved with changes, never revised | Approved with conditions; the corrected file was made later | A later version carrying the requested change |
| Newest export of an older cut | Most recent modified date, older edit | Lineage IDs and content hash match the superseded version |
| Rendition passed off as a version | Different format or size of the approved cut | Different DocumentID with a parent link to the approved file |
| Localized variant | Approved cut with another language track or end card | Territory in the rights record and a separate decision for the variant |
| Expired or wrong-channel rights | Approved file whose music or talent window ended, or cleared for another channel | Rights constraint dates and channel against the question |
| Shipped without approval | Appears in the delivery log | No approving decision on that version |
How should the benchmark score an agent's answer?
Score the pick, the evidence and abstention separately, then report each by decoy family. A single accuracy figure hides the costly error, which is choosing a rejected or out-of-window file with confidence. Decide in writing, before running anything, how much worse a wrong pick is than a refusal for your use case.
Run one ablation alongside the main test: the same items with the original filenames restored. If accuracy jumps when names are visible, the agent is still leaning on them, and the size of the jump is a useful number to track between agent versions. If accuracy falls, the names in your pool are actively misleading, which is common in folders where final was typed on every export.
Keep a small set of items where the filename happens to be right and the evidence agrees. An agent that refuses to trust anything and abstains everywhere will look cautious on hard items and fail these, which keeps abstention from becoming a free strategy.
- Selection: the chosen file ID equals gold_selection.
- Evidence: the agent cites the review decision on the gold version, plus the rights record when the item turns on rights; citing only lineage or a status field counts as weak evidence.
- Abstention: declining on a none_cleared item is correct; declining when a cleared file exists is an error.
- Severity: rejected, superseded and out-of-window picks are reported apart from other misses.
- Splits: decoy family, number of candidates, whether lineage metadata was present, and whether rights decided the item.
Illustrative item: three cuts of a spot and a lapsed music license
This item is invented for illustration and does not describe any company's files. A brand's spot exists as three candidates. File A is the 30-second cut whose review shows Approved by the client lead on version 3. File B is version 4 of the same cut, made after a reviewer marked version 3 Approved with Changes and asked for a revised legal line; version 4 was uploaded but never decided. File C is a vertical rendition of version 3 for social, with a different DocumentID and a parent link to File A, and it was the most recently modified file in the folder.
The question asks which file is approved for broadcast in the United States on a date inside the music license window. The gold answer is File A, citing the version 3 decision. File B is newer but has no decision; File C is a rendition cleared for social only. The exception: move the question date to a day after the music license ends and the gold answer becomes none_cleared, even though File A is still approved, because the rights record says the track may no longer run. An agent that answers File A on that date picked an approved file the business cannot use, which is exactly the error the severity split exists to surface.
Items to exclude, and when a rights reviewer sets the label
Leave out jobs where approval happened only in email, a call or a meeting and was never written into a system the labeler can show, unless the item is built to test abstention. Leave out jobs with more than one simultaneously valid version for the same use, such as A/B test variants both approved, or narrow the question until only one answer is correct.
Send the label to a rights reviewer whenever the answer depends on reading a contract: talent usage terms, music sync windows, stock license scope or a client's master services agreement. The benchmark records the reviewer's reading; it does not ask the agent or the labeler to interpret the contract. State the limits on the results page too. A benchmark built from one agency's jobs measures that agency's review habits, proofing tools label decisions differently, and an agent that scores well on video may still miss approval cues on print layouts.
Licensing review histories as benchmark material
The raw material for this benchmark is a creative business's own job history: every version uploaded for review, the decisions on each, the rights records and the delivery log. Agencies, studios and in-house creative teams often keep years of it. Holding those files is not the same as being free to license them; client ownership of the work, talent and music terms and confidentiality clauses each need review, and removing names does not create a right to license.
For a studio or agency holding that history, SourceX can carry approved job records through rights review, contracting, preparation and delivery to AI developers. A creative business starts with a brief company assessment that asks for no files, and no proofing record is handed over until it has approved the scope and terms in a signed agreement.
Frequently asked questions
Why not let the agent use filenames at all?
Because filenames are typed by people and copied freely, so a name containing final says nothing reliable about approval. Keep the original names in a sealed column for auditing, and run a separate ablation with names visible if you want to measure how much an agent leans on them.
Does XMP or Content Credentials metadata record who approved a file?
Not in the fields their specifications define for versions and lineage. XMP Media Management identifiers and history describe versions, renditions and actions such as edited or published, and C2PA ingredients describe parent and component relationships. Approval decisions live in the review or proofing tool, so store them separately.
How should Approved with Changes be labeled?
Label it as its own state and check whether a later version carries the requested change. If it does and that version has a decision, the later version is usually the answer; if no corrected file exists, decide in the labeling guide whether the conditionally approved version counts, and apply that rule to every item.
How many candidate files should each item have?
Enough to include at least one decoy from the families you are testing, which in practice means two to eight. Record the count on every item so results can be split by pool size, since larger pools are harder.
Who should label the gold answers?
Two people who know the review tool label each item independently, and an adjudicator resolves disagreements. Any item whose answer turns on a contract goes to a rights reviewer, whose reading is recorded with the item.
Sources
- XMP Media Management defines DocumentID for all versions and renditions, InstanceID per save, VersionID and a version history that may be incomplete. Source
- XMP history events record actions such as converted, copied, created, edited, published and saved, each with a timestamp. Source
- xmpRights:Marked is omitted when the rights state is unknown, and XMP rights properties express rights rather than enforce them. Source
- C2PA 2.2 ingredients carry a relationship of parentOf, inputTo or componentOf, with at most one parentOf ingredient per manifest. Source
- Workfront proof statuses are Pending, Approved, Approved with Changes, Changes Required and Not Relevant, and overall status follows the worst-case participant. Source
- IPTC Photo Metadata includes Model Release Status and a Data Mining property added in 2023.1; an empty Data Mining value does not grant permission. Source
- EIDR models creative versions as Edits and technical encodings as Manifestations; related-content relationships support distribution and rights systems. Source
- Workfront Proof administrators can rename and hide proof decision options, and a rename updates the decision name on existing proofs. Source
See if your company qualifies
A short company assessment. No data uploads are needed.