Systems and records
Dataset encoding validation: what blank counts miss in 618 files
By SourceX Editorial · Updated
10 sections
Short answer
Validate encoding first, because broken decoding rarely creates blank cells. It damages text that still counts as present and hides columns from lookups. In 618 CSV and TSV files from FiveThirtyEight's public repository, reading everything as UTF-8 with errors ignored altered 102,229 cells, added no blanks and left 8,163 cells under headers that no longer matched. Separate recoverable mis-decoding from characters lost before export, then count.
Checklist
Dataset encoding validation
Key takeaways
- In a 618-file public corpus, lenient UTF-8 decoding altered 102,229 cells and created zero new blank cells, so the blank count could not see the damage.
- Missing text has three sizes: missing columns, blank cells and missing characters; encoding failures mostly hit the first and the last.
- A byte-order mark and lenient decoding of legacy headers left 8,163 cells (2,718 plus 5,445) under column names that no longer match the documented ones.
- A successful decode is not proof of the right codec: one byte read as E-circumflex under Windows-1252 and as a no-break space under Mac Roman, 286 times in one file.
- Replacement characters and question marks in place of letters mean loss before export; ask for a re-export instead of trying to repair the file.
Why validate dataset encoding before counting missing text?
Because encoding failures rarely create blank cells; they damage text that still counts as present and hide whole columns from lookups. In a 618-file audit of FiveThirtyEight's public data repository, reading every file as UTF-8 with errors ignored altered 102,229 cells and created zero new blanks. A missing-text count taken at that point would have looked identical to the clean one.
Dataset encoding validation is the step that decides which bytes became which characters before any completeness metric is computed. The decision it supports is a delivery decision: accept the file, repair it with a declared conversion, or return it to the supplier because something was lost before it was exported.
Missing text comes in three sizes, and each needs its own count. A missing column is a field the consumer cannot find by name. A blank cell is a field present but empty. Missing characters are letters, accents or punctuation replaced or dropped inside a non-blank value. A completeness report that only counts blank cells sees the middle size and none of the others.
What 618 public CSV and TSV files showed about decoding
The audit read every .csv and .tsv file in the fivethirtyeight/data repository at commit 4c1ff5e (committed February 25, 2025), a public collection of the data behind FiveThirtyEight articles, published under the Creative Commons Attribution 4.0 license. Of 620 such paths, two are Git LFS pointer files rather than data, leaving 618 files and 38,828,026 data cells.
Most of the corpus is plain: 541 files contain only ASCII bytes. But 24 files are not valid UTF-8, nine begin with a UTF-8 byte-order mark, and none of the repository's 174 README files names a character encoding. A consumer receiving these files has to infer how they were written, which is exactly the situation a delivery manifest should prevent.
The table compares four reading policies on the same bytes. The blank share moves only when a policy drops files, while the damage a lenient reader does never reaches the blank count at all:
| Reading policy | Files read | Blank cells (share) | What the blank count misses |
|---|---|---|---|
| Strict UTF-8, skip files that fail | 594 of 618 | 1,324,293 of 35,132,343 (3.77%) | 24 files and 3,695,683 cells never read; 2,718 first-column cells sit under a header that starts with an invisible BOM character |
| UTF-8 with errors ignored, BOM kept | 618 of 618 | 1,421,776 of 38,828,026 (3.66%) | 102,229 cells silently altered; 8,163 cells under headers that no longer match their names |
| Validated: BOM stripped, legacy files decoded with a named codec | 618 of 618 | 1,421,776 of 38,828,026 (3.66%) | Upstream loss the decoder cannot undo: 1 cell with U+FFFD, 1,763 cells with a question mark where an apostrophe was |
| Same, but the 24 legacy files read as Mac Roman instead of Windows-1252 | 618 of 618 | 1,421,776 of 38,828,026 (3.66%) | Nothing in the count changes, yet bytes such as 0xCA become different characters: the codec choice is invisible to completeness metrics |
Extraction loss, mis-decoding or normalization: which one is in the file?
Each failure class leaves a different fingerprint, and only one of them means information is gone. Mis-decoding keeps every byte, so the original text can be recovered once the right codec is known. Normalization differences keep every character, written in a different but canonically equivalent sequence. Extraction loss happened before the file was written: a character was replaced, and the file now carries the replacement.
The corpus has all three. The 24 invalid files are mis-decoding risks: their bytes are intact and decode cleanly once a legacy codec is chosen. Three valid UTF-8 files hold 964 cells of reversible mojibake, text that was decoded with the wrong codec at some earlier step and saved again (the README of the largest case says its data was copied from a Justice Department web page), so a curly apostrophe now appears as a three-character sequence such as a-circumflex, euro sign, trademark sign. Re-encoding such a cell as Windows-1252 and decoding the result as UTF-8 reverses one wrong decode, and the audit counts a cell when that round trip succeeds and changes it.
Extraction loss looks different. One cell in a 2015 Thanksgiving survey file ends in U+FFFD, the Unicode replacement character, where a write-in answer appears to have had an accented letter; the file is valid UTF-8, and the original letter is not in it. Two files from a masculinity survey contain 1,763 cells where a question mark sits between a letter and a contraction ending, as in isn?t, and a classic rock file writes band names with question marks in place of umlauts. The masculinity files are plain ASCII and the classic rock file is valid UTF-8, so a byte-level validator passes all three.
Normalization was the one class with no findings: no cell in the 618 files changes under Unicode NFC normalization. The W3C's character model explains why the check still belongs in the suite: canonically equivalent sequences should look the same and be treated as identical, yet they are different code points, so equality tests, deduplication and joins split them unless both sides are normalized.
Why a decoder that succeeds can still be wrong
A successful decode proves only that the bytes are legal in that codec. Python's codec documentation says it plainly: without external information it is impossible to reliably determine which encoding was used, and each charmap encoding can decode any random byte sequence. UTF-8 is the exception, because its byte structure rejects arbitrary sequences.
The audit shows what that means in practice. Windows-1252 decoded 17 of the 24 invalid files without error, yet in a food survey file the most common non-ASCII byte, 0xCA, appears 286 times. Windows-1252 reads it as a capital E with circumflex; Mac Roman, the older Macintosh encoding, reads it as a no-break space. Only one of those is plausible in survey answers, and no decoder error would tell you which. Apostrophes give the codec away. Between two letters, where an apostrophe usually sits, two of the invalid files use byte 0x92, the Windows-1252 right single quotation mark, eight times in all. Four other files use byte 0xD5 there, eight times as well; Windows-1252 renders it as a capital O with tilde and Mac Roman as a right single quotation mark. No file uses both, so one repository holds at least two legacy encodings, and nothing in the files says which is which.
Some files are wrong twice. A file of Tenth Circuit appeals cases repeats the bytes 89 DB D2 six times; under Mac Roman they read as a-circumflex, euro sign, left double quotation mark, which is itself the Windows-1252 rendering of a UTF-8 en dash. That text was mis-decoded once, then saved in a second legacy encoding. Repairing it means reversing both steps in order, and a validator that stops at a successful decode never finds out.
Labels add a further trap. The WHATWG Encoding Standard, which browsers follow, treats latin1 and iso-8859-1 as labels for windows-1252, while other software maps the bytes 0x80 to 0x9F differently. Seven of the invalid files contain bytes that Python's Windows-1252 codec leaves undefined, so the audit's fallback decoded them as ISO-8859-1, which turns every byte from 0x80 to 0x9F into an invisible control character. A manifest that says latin1 has therefore not settled the question.
How a byte-order mark or a damaged header becomes a missing column
The missing text a schema check sees came from headers, not cells. Nine files start with the three bytes EF BB BF. A reader that strips them, such as Python's utf-8-sig codec, sees normal headers. A plain UTF-8 reader keeps them as an invisible U+FEFF at the start of the first header, so a lookup by the documented column name fails, and a completeness check reports all 2,718 cells in those first columns as absent.
Lenient decoding does the same to headers that carry legacy bytes. Under errors ignored, 20 header fields in seven files changed, and 5,445 data cells sit beneath them. Two cases show how quiet this is: a Star Wars survey file and a survey on couples who sleep in separate beds each contain invalid bytes only in one header. Not a single data cell changes, yet 1,187 and 1,094 cells disappear from any lookup by the original column name.
Neither effect reaches a blank-cell count, because the cells still hold values. Both reach any pipeline that selects columns by name, which is most of them. The standards do not settle the BOM question for you: Python's documentation says the BOM is discouraged in UTF-8 and should generally be avoided, while the Unicode Standard's current core specification says it does not recommend for or against a BOM in UTF-8 data. That is why the policy belongs in the manifest rather than in an assumption.
The encoding acceptance test matrix, check by check
Run these checks in order on the raw bytes, before parsing, profiling or counting blanks. Each row names the test as the audit ran it, what it separates, whether the damage can be undone, and what it found at commit 4c1ff5e. Every check reads files only, so it can run on a supplier sample or a full delivery:
| Check | Test as run | What it separates | Recoverable? | Found in 618 files |
|---|---|---|---|---|
| Strict UTF-8 | Decode bytes after any BOM with errors raising; count invalid sequences | UTF-8 from legacy or mixed bytes | Yes, once the codec is declared | 24 files, 286,102 invalid sequences |
| Codec context | Read invalid files as Windows-1252 and Mac Roman; list frequent non-ASCII byte runs and bytes between letters with both readings | Which legacy codec fits the content | Yes, if context or a declaration decides | 17 decode as Windows-1252; apostrophe byte 0x92 in 2 files, 0xD5 in 4 |
| Byte-order mark | First bytes EF BB BF, FF FE or FE FF; compare first header with and without U+FEFF | A signature from a missing column | Yes, strip and record | 9 files; 2,718 first-column cells |
| Header round trip | Compare headers under the chosen codec and a lenient UTF-8 read; flag any that differ | Renamed columns from intact ones | Yes, with the right codec | 20 headers in 7 files; 5,445 cells beneath |
| Reversible mojibake | Re-encode non-ASCII cells as Windows-1252, decode as UTF-8; flag cells that change | Double-decoded text from genuine accents | Yes, by reversing the steps | 964 cells in 3 valid UTF-8 files |
| Replacement character | Count U+FFFD in decoded valid UTF-8 | Loss before export from a decoder artifact | No: the original is gone | 1 cell |
| Question-mark substitution | Letter, question mark, letter, excluding URLs and query strings; apostrophe subset | Lossy export from real punctuation | No, unless the source system re-exports | 1,788 candidate cells in 5 files; 1,763 apostrophe cells in 2 |
| NFC stability | normalize('NFC', cell) differs from cell | Equivalent but different code points | Yes, normalize both sides | 0 cells |
What the delivery manifest should record about text
A manifest closes the gap the audit exposed: none of the 174 READMEs said how the files were encoded, so every decision above had to be inferred. The fields below are the encoding-specific part of a versioned delivery manifest; the CSV dialect, null tokens and checksums are covered in the pages linked at the end, and they matter as much. Record these per file and per delivery version:
- Encoding name as a WHATWG label (utf-8, windows-1252, macintosh), plus the source system's own setting where the export tool exposes one.
- Byte-order mark: present or absent, and whether the consumer should strip it.
- Normalization form applied to text fields (NFC, or none), and the step that applied it.
- Conversion history: every re-encoding between the source system and the file, in order, with the error mode used at each step (fatal or replacement), so double-decoding can be reversed instead of guessed.
- Counts at export: invalid sequences, U+FFFD characters and question-mark substitutions, measured by the supplier with the same rules the buyer will rerun.
- Header names as UTF-8 strings with their byte length, so a renamed column fails a comparison instead of a downstream join.
- The acceptance result for each check in the matrix, with the tool version and the date it ran.
When is a file with lost characters still acceptable?
Sometimes, if the loss is small, flagged and irrelevant to the task. A loadable file or a valid checksum can still carry broken meaning: the checksum proves the bytes arrived as sent, not that they were right when sent. The acceptance decision therefore turns on where the lost characters are and what the data will train or evaluate.
A single replacement character in a free-text survey answer is unlikely to matter for most uses; flag the cell and move on. Apostrophes replaced by question marks in 1,763 cells of answer text are different if the task involves tone, negation or quoted speech, because isn?t no longer reads as a negation to a tokenizer. Names with question marks in place of letters break entity matching and deduplication. Ask the supplier to re-export from the source system with a declared encoding, since the characters cannot be rebuilt from the file.
The strongest objection is that some of these flags are real. A question mark between letters can be genuine punctuation in casual text, and the audit's candidate rule already excludes URLs and query strings because they produced false matches. Treat each check as a triage step that routes cells to review, and report the counts with the rule that produced them rather than as defects.
The limits of this audit follow from its design. It used one public repository, chose Windows-1252 as the first fallback, and did not decide which legacy codec is correct for each file; it measured what each reading policy does. Business-system exports from ticketing, CRM or ERP tools will show different mixes, which is why the checks, not these counts, are the reusable part.
Methods note: the 618-file encoding audit
Corpus: every path ending in .csv or .tsv, case-insensitive, in git ls-tree of fivethirtyeight/data at commit 4c1ff5e3aef1816ae04af63218015066e186c147, read from a sparse checkout. Two Git LFS pointer files (science-giving/science_federal_giving.csv and scrabble-games/scrabble_games.csv) were excluded. The script, written for this article on 2026-10-28, reads files only and runs no code from the repository.
Byte rules: a UTF-8 BOM is EF BB BF at the start; ASCII-only means every byte after any BOM is below 0x80; valid UTF-8 means Python 3's strict decoder succeeds on the bytes after any BOM; invalid sequences are the number of error callbacks Python's UTF-8 decoder raises. The reference decode is UTF-8 when valid, otherwise strict Windows-1252 (Python cp1252), otherwise ISO-8859-1. Parsing uses Python's csv module with a comma for .csv and a tab for .tsv, on text wrapped with newline set to the empty string as the csv documentation asks; row 1 is the header and every field of later rows is a data cell. A blank cell is empty after Python's str.strip(), which also removes no-break spaces.
Cell rules: a U+FFFD cell contains the replacement character in the reference decode; a reversible mojibake cell has a non-ASCII character and changes when encoded as Windows-1252 and decoded as strict UTF-8; an NFC-unstable cell differs from its NFC form. Question-mark candidates contain an ASCII letter, a question mark and an ASCII letter in a row, and contain neither :// nor a question mark followed by a run of letters, digits, underscores, dots or hyphens and an equals sign; the apostrophe subset also contains an ASCII letter, a question mark and a lowercase s, t, re, ll, ve, d or m ending at a word boundary. Mojibake counts are reported for valid UTF-8 files only, where the reference decode is the file's own encoding; the U+FFFD, question-mark and NFC counts cover all 618 files.
Policy rules: the lenient decode applies errors ignored to the 24 invalid files and is compared cell by cell with the reference decode (row and field counts were asserted equal); errors replaced puts U+FFFD in 102,183 of the 102,229 altered cells, and the other 46 hold byte sequences that are valid UTF-8 inside otherwise legacy files. Header damage counts data cells under a header field that differs between the two decodes. The BOM effect reads each BOM file with plain UTF-8, confirms the first header then starts with U+FEFF, and counts the data rows with at least one field, which are the cells under that header. A Mac Roman decode of the same 24 files gave the same 97,483 blank cells. Byte readings come from the three most frequent runs of bytes at 0x80 or above in each invalid file, and from letter-flanked bytes: single bytes at 0x80 or above whose neighbors are both ASCII letters, with 0x92 and 0xD5 counted for every invalid file. Shares are exact fractions rounded half-up to two decimals. README.md files were searched, case-insensitively and as whole words, for utf-8 or utf8, encoding, encoded, latin-1 or latin1, iso-8859 or iso8859 with an optional part number, windows-1252 or windows1252, cp1252 and charset; none matched. Not measured: which legacy codec each author used, files outside .csv and .tsv, and any meaning lost before the source system stored the text. Outputs contain counts only, no cell values.
Putting encoding checks into a data request
Write the checks into the request, not the post-delivery review. Name the encoding and BOM policy you will accept, attach the matrix as the acceptance test, and ask the supplier to report its export-time counts with the same rules, so both sides measure the same thing.
SourceX sources datasets on request rather than holding them in stock, and its request form asks buyers for format, language, de-identification requirements and additional specifications, which is where an encoding requirement and its acceptance test fit. SourceX licenses the data from the company that holds it and licenses it to the buyer in its own name, so a test stated in the request shapes what the supplier is asked to prepare.
Frequently asked questions
Is passing a UTF-8 check enough to accept a text dataset?
No. A strict UTF-8 check catches undeclared legacy bytes, but files that already lost characters before export pass it: in the 618-file audit, 1,763 cells with question marks in place of apostrophes sat in valid ASCII files. Run the replacement-character, mojibake and substitution checks as well.
Can mojibake be repaired automatically?
Often, when the steps that produced it are known. Text decoded once with the wrong codec can be restored by re-encoding with that codec and decoding as UTF-8; text mis-decoded twice needs both steps reversed in order. A conversion history in the manifest turns that from guesswork into a procedure.
Should a byte-order mark fail a delivery?
Not by itself, but it should be declared and stripped on purpose. In the audit, a BOM left in place made 2,718 first-column cells unreachable by their documented names. Python's documentation discourages the BOM in UTF-8, and the Unicode Standard takes no position, so the manifest should state the policy.
What does U+FFFD in a delivered file mean?
The replacement character means some decoder met bytes it could not map and substituted U+FFFD. If the delivered file is valid UTF-8 and contains U+FFFD, the substitution happened before export and the original character is not recoverable from the file. Ask for a re-export from the source system.
Does Unicode normalization change missing-value counts?
Rarely, because normalized and unnormalized text are both non-blank. It changes equality: canonically equivalent strings written with different code points fail exact matches, joins and deduplication. The audit found no NFC-unstable cells, but the check is cheap and belongs in every delivery test.
Sources
- The fivethirtyeight/data repository at commit 4c1ff5e holds the data behind FiveThirtyEight articles, including 620 .csv and .tsv paths, two of them Git LFS pointers, and 174 README files. Source
- The repository's LICENSE file is the Creative Commons Attribution 4.0 International license. Source
- The WHATWG Encoding Standard calls UTF-8 the most appropriate encoding for interchange of Unicode, defines replacement and fatal error modes, sniffs EF BB BF as a UTF-8 BOM, and treats latin1 and iso-8859-1 as labels for windows-1252. Source
- The W3C Character Model for the World Wide Web: String Matching says canonically equivalent sequences ideally look the same and should be treated and processed as if identical. Source
- Python's codecs documentation says that without external information it is impossible to reliably determine which encoding was used, describes the ignore and replace error handlers, and says the BOM in UTF-8 is discouraged and utf-8-sig skips it. Source
- Python's csv documentation says a file object passed to the reader should be opened with newline set to the empty string. Source
- Python's unicodedata documentation defines normalize(form, unistr) for the NFC, NFKC, NFD and NFKD forms. Source
- The Unicode Standard's core specification, chapter 2, says the standard does not recommend for or against use of a BOM in UTF-8 data. Source
- SourceX's request form says datasets are sourced on request, not held in stock, and asks for format, language, de-identification requirements and additional specifications; SourceX licenses data to the buyer in its own name. Source
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.