Skip to content

Systems and records

Dataset multilingual metadata standard: where language labels belong

By SourceX Editorial · Updated

9 sections

Short answer

Put language labels at every level a tool reads: the dataset card and Croissant or DCAT metadata for discovery, each file's manifest entry, each record, and each field or span that differs from its record. Use BCP 47 tags throughout, require every lower-level tag to appear in the list above it, and record whether each tag was declared by the source, detected or verified by a person.

Countries and languages

Dataset multilingual

This page covers dataset multilingual metadata standard: where language labels belong.Illustrative

Key takeaways

  • Dataset-level language lists serve discovery; training and evaluation pipelines need record, field and span labels.
  • BCP 47 tags work at every level, and most dataset metadata standards already expect them.
  • Record how each tag was assigned, so buyers can separate source-declared, detected, verified and translated text.
  • Labels inside the data files survive copying and sampling better than labels in a sidecar alone.
  • Containment, syntax and registry checks catch most labeling faults without the supplier's systems.

Where should language labels appear for downstream dataset tools to use them?

At every level a tool actually reads: the dataset card, the machine-readable dataset metadata, each file or distribution, each record, and each text field or span that can differ from its record. A dataset multilingual metadata standard that stops at the first level lets people find the dataset but leaves training and evaluation pipelines guessing which rows are in which language.

Use one tag format everywhere, BCP 47 language tags, and make the levels agree: every tag found at a lower level must appear in the list above it. Then say how each record-level tag was assigned, because a tag declared by the source system, one detected by a model and one confirmed by a person are not equally trustworthy.

The rest of this page gives a record, field and case language schema with a valid record, a failing record and the checks a buyer can run without access to the supplier's systems.

Why does a dataset-level language list fail a training pipeline?

Because the common dataset-level fields are built for discovery, not filtering. The Hugging Face dataset card's language field is a list used, among other things, to let people filter and find datasets. Croissant's inLanguage is a dataset property describing the languages of the dataset's content. Neither tells a data loader which rows to keep for a Spanish-only fine-tune.

Business records make this worse. A support case can have an English subject line, a Mexican Spanish body, an agent reply in English and a Brazilian Portuguese attachment. A dataset-level list of en, es and pt is accurate and useless for a pipeline that needs the Spanish customer turns only. A record-level tag of es is wrong for half of that case.

DCAT 3 offers a partial answer at the distribution level: when a dataset is available in separate language versions, it recommends one distribution per language with a single language value, while the dataset carries all of them. That works for translated document sets. It does not help when languages are mixed inside one record, which is the normal state of operational text.

Which tag format should every level use?

BCP 47 language tags, as defined in RFC 5646, at every level. A tag is a sequence of subtags separated by hyphens, each at most eight ASCII letters or digits, and the conventional form writes the language in lowercase, a four-letter script in titlecase and a two-letter region in uppercase, as in es-MX, pt-BR or sr-Latn. Matching is case-insensitive, but writing the conventional form avoids needless mismatches in naive string comparisons.

Add only the subtags that carry information the buyer will use. A script subtag matters when one language is routinely written in more than one script, as with Serbian in Latin or Cyrillic, or Chinese in simplified or traditional characters; a tokenizer or evaluation set built for one script is not interchangeable with the other. A region subtag matters when vocabulary, spelling or regulation differs by country, as it often does for Spanish, Portuguese and French customer text. A bare language subtag is better than a region guessed from the customer's billing address.

Most dataset tools already expect BCP 47, with one exception worth mapping explicitly. The table shows what each common target reads.

Which tag format should every level use?
TargetFieldLevelExpected value
Hugging Face dataset cardlanguageDatasetLanguage codes; the template's examples are two-letter codes such as fr and en
Hugging Face dataset cardlanguage_detailsDatasetBCP 47 tags with region, such as fr-FR and en-US
Croissant 1.1inLanguageDatasetLanguage or text values for the languages of the dataset content
schema.orginLanguageAny creative workA code from the IETF BCP 47 standard
DCAT 3dcterms:languageDataset and distributionOne value per language distribution, several on the dataset
JSON-LD 1.1@language and @directionAny string value or a default for a document or subtreeA well-formed BCP 47 tag, and ltr or rtl for base direction
Apache Arrow and Parquet via Arrowcustom_metadataSchema and each fieldUser-defined key-value pairs, so a field can carry its own language key

What should the record field artifact and case language schema include?

Eleven fields across five levels. The schema is versioned, every identifier is stable, and every language value is a BCP 47 tag. Fields marked required must be present on every record; the others are required when the condition applies.

At a glance

What should the record field artifact and case language schema include

Keep language assignment separate from language value. The lang_method field is what lets a buyer exclude machine-detected tags from an evaluation set, or audit a sample of them.

What should the record field artifact and case language schema include?
FieldLevelRuleExample (illustrative)
schema_versionPackageRequired; semantic version of this schema1.2.0
dataset_languagesPackageRequired; every tag used below must appear hereen-US, es-MX, pt-BR
file_languagesFile (manifest entry)Required; tags present in the file, with file path relative to the package root and a hashcases/part-0003.parquet: en-US, es-MX
record_id, case_idRecordRequired; stable across releasesrec-00412, case-00187
record_langRecordRequired; dominant language, or mul when no language dominateses-MX
lang_methodRecord and fieldRequired; one of source_declared, detected, human_verified, translateddetected
field_langsFieldRequired when any text field differs from record_langsubject: en-US; body: es-MX
spansSegmentRequired for code-switched text; start, end and tag per span, using character offsetsbody 0 to 118: es-MX; 119 to 160: en-US
translation_ofRecordRequired when lang_method is translated; points to the original record_idrec-00398
directionFieldRequired for right-to-left text; ltr or rtlrtl for an Arabic body field

Worked example (illustrative): a support case with four languages

This case is invented to show the schema; it is not a SourceX record. A distributor's help desk case contains a customer email in Mexican Spanish with an English subject line, an agent reply in US English, and a product sheet attachment in Brazilian Portuguese that was machine-translated into English for the agent.

The valid encoding has four records under one case_id. The customer email is record_lang es-MX with field_langs subject en-US and body es-MX, lang_method source_declared because the help desk captured the customer's language preference, and a span list because the body ends with a pasted English error message. The agent reply is en-US, source_declared. The attachment is pt-BR, detected, with its own record. The machine translation is a fourth record, en-US, lang_method translated, with translation_of pointing to the attachment.

The dataset card lists en, es and pt under language and en-US, es-MX and pt-BR under language_details. The Croissant file lists the same three tags in inLanguage. The Parquet file for cases stores record_lang as a column and carries the same three tags in its schema-level custom metadata, so a reader can check the file without opening the manifest.

A pipeline that wants human-written Spanish customer turns now has a precise filter: record_lang or field language es-MX, lang_method not translated, role customer. Without the field and span levels it would have kept the English error message and dropped nothing.

What does a failing record look like, and which checks catch it?

A failing record usually looks plausible. Here is an illustrative one: record_lang sp, field_langs body en_US, lang_method missing, a translated record with no translation_of, and a span tagged fr-CA in a file whose manifest lists only en-US and es-MX. Each fault breaks a different downstream tool, and each is caught by a mechanical check.

Do not let the validator repair records silently. Rewriting sp to es or en_US to en-US is usually right, but an automatic fix hides how often the source system produced bad tags, which is itself evidence about the reliability of every other tag from that system. Quarantine failing records, report counts per check and per source system, and apply corrections as a documented, versioned step that the buyer can see in the delivery notes.

  • Well-formed tags. Every tag parses under RFC 5646 syntax; en_US fails because subtags are separated by hyphens, not underscores.
  • Registered subtags. Every language, script and region subtag exists in the IANA Language Subtag Registry; sp fails because Spanish is es.
  • Containment. Every span tag appears in its field, record, file and dataset lists; the fr-CA span fails the file and dataset checks.
  • Method present. Every record and field-level tag has a lang_method value.
  • Translation link. Every translated record names an existing translation_of record in a different language.
  • Span coverage. Spans in a field do not overlap and stay within the field's length.
  • Direction. Fields in scripts written right to left carry direction rtl.
  • Card agreement. The Hugging Face language list, language_details, Croissant inLanguage and dataset_languages describe the same set once tags are reduced to their language subtag.

Can the buyer check every language label using only the delivered files?

Yes, if the package carries its own manifest, schema and rules. The manifest lists every file by relative path with a hash and its file_languages; the schema file defines the eleven fields and their allowed values; and the checks above are declarative, so a buyer can implement them in any language against the files alone.

Put the record-level labels inside the data files as columns, not only in a sidecar, so that they survive when files are copied, sampled or converted. Arrow-based formats let each field carry user-defined metadata, which is a practical place to repeat the field's language set for tools that read schemas but not manifests. For linked-data exports, JSON-LD can tag individual string values with a language and a base direction, and a default language can be set for a whole document or subtree.

Version the schema separately from the content. A buyer comparing two deliveries needs to know whether a change in language counts came from new records or from a new detection method, and only schema_version together with lang_method can tell them apart.

Which language decisions need an engineer and a linguist before delivery?

Three choices change downstream results enough that the buyer's integration owner and a language reviewer should agree on them in writing before records are labeled.

First, the threshold for mul: at what share of a second language does a record stop having one dominant language. Second, the detector and its version for any detected tags, plus the size of the human-verified sample used to estimate its error rate. Third, how dialect and region are handled: whether es-MX and es-US are kept apart, and whether a region subtag records where the text was written or where the speaker is from.

Record each decision in the dataset card's free-text section and in the schema file. A model trained on region-tagged text inherits those definitions, so they belong with the data rather than in an email thread.

Agree, too, on what happens when the supplier's systems simply do not know. Many ticketing, CRM and order systems store a customer's preferred language as a profile setting that may be years old, or a site default that was never changed. Treat these as source_declared only when the field was set at the time of the message; otherwise mark the tag detected or leave it at the language subtag alone, and say so in the card.

Requesting language-labeled business records

Buyers who need multilingual operational records can make the schema part of the request, so the supplier labels to it from the start instead of retrofitting tags after delivery.

SourceX sources datasets on request from U.S. businesses rather than holding stock; buyers describe scope, volume, language, format, timing and licensing needs in the request-data form. De-identification is performed by SourceX through approved processing routes, which is the stage where field and span labels should be preserved.

Frequently asked questions

What is the standard format for language labels in a dataset?

BCP 47 language tags, defined in RFC 5646, such as en-US, es-MX or sr-Latn. schema.org's inLanguage asks for BCP 47 codes, and JSON-LD requires well-formed BCP 47 tags for language-tagged strings, so one format can serve every level.

Does the Hugging Face dataset card support regional language tags?

Yes. The card's language field takes language codes, with two-letter examples, while the language_details field takes BCP 47 tags with regions, such as fr-FR. List both so discovery filters and regional detail agree.

How should a dataset label records that mix languages?

Give the record a dominant language, or mul when none dominates, add field-level tags where fields differ, and add character-offset spans for code-switched text. Record how each tag was assigned so buyers can separate source-declared from detected labels.

Where should language labels live in Parquet files?

As columns for record and field tags, so they survive copying and sampling, with the file's language set repeated in schema or field metadata. Arrow-based formats allow user-defined key-value metadata on the schema and on each field.

Should machine-translated records share a language tag with originals?

They can share the tag of the language they are written in, but they need lang_method set to translated and a link to the original record. Without both, a buyer cannot exclude translations from evaluation sets or check them against their sources.

Sources

  • RFC 5646 defines BCP 47 language tags as hyphen-separated subtags of up to eight ASCII characters, with a conventional case format for language, script and region subtags. Source
  • The Hugging Face dataset card template has a language list with two-letter examples and a language_details list for BCP 47 tags such as fr-FR and en-US. Source
  • Hugging Face documentation says dataset card metadata such as language helps users filter and discover datasets on the Hub. Source
  • Croissant 1.1 defines inLanguage as a dataset property giving the language or languages of the dataset content. Source
  • schema.org's inLanguage property asks for a language code from the IETF BCP 47 standard. Source
  • DCAT 3 recommends one distribution per language, each with a single dcterms:language value, when a dataset is available in separate language versions. Source
  • JSON-LD 1.1 lets string values carry a BCP 47 language tag and a base direction, with a default language settable for a document or subtree. Source
  • Apache Arrow's schema format gives both the schema and each field user-defined key-value metadata. Source

Tell us what data you need

Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.

See if your company qualifies