Calibration Requirements: Intrinsics, Extrinsics and Hand-Eye

The buyer's problem

Raw frames without calibration are hard to use for anything geometric. Multi-view fusion, depth alignment, 3D hand or object tracking, and mapping image pixels to robot actions all depend on knowing the lens model and the sensor poses. Datasets often arrive with calibration missing, stale or described in free text.

DROID is a useful reference for what a documented rig provides: the dataset ships camera calibration, depth and language instructions with each episode, recorded on a Franka Panda arm with two exterior stereo cameras and a wrist-mounted camera, and the same rig was replicated at 13 institutions (Khazatsky et al., 2024). That consistency is a design choice that a buyer can ask any collector to match.

Three layers of calibration

Intrinsics. These describe each camera: focal length, principal point, lens distortion. A widely cited technique observes a planar pattern at a few orientations (at least two), does not require the motion to be known, models radial distortion, and finishes with a maximum-likelihood refinement (Zhang, 2000). Check that the lens model named in the metadata, pinhole, fisheye or other, matches the one used to estimate the numbers. Rolling-shutter cameras need extra care.

Extrinsics. These give the pose of each camera relative to a reference frame, whether another camera, the robot base or the world. For multi-camera rigs the reference should be stated in the metadata, along with axis conventions and units.

Hand-eye. This is the transform between a camera and a robot link. For a wrist camera (eye-in-hand) it is the fixed pose of the camera relative to the flange or gripper; for a fixed camera (eye-to-hand) it is the pose of the camera relative to the robot base. It is usually estimated by moving the arm through several poses while the camera observes a target, combined with the arm's forward kinematics. Errors here move the apparent position of objects relative to the gripper, and policies learn the wrong offsets.

Camera-to-IMU transforms and time offsets form a fourth layer when inertial sensors are present. Kalibr is an open toolbox that handles multi-camera intrinsics and extrinsics, spatial and temporal camera-IMU calibration and rolling-shutter intrinsics, and writes a PDF report, a text summary and YAML (ETH Zurich ASL, Kalibr). Whatever tool a provider uses, ask for its output report, not just the final numbers. Temporal offsets tie into sensor synchronization requirements.

What calibration data cannot do

Calibration metadata describes geometry, not task content. It does not supply robot actions, forces or labels. Nor can it be recovered reliably after the fact from workplace video: archive footage from handheld, head-mounted or ceiling cameras usually lacks calibration targets, lens details and mount records, so geometric uses are limited to what self-calibration can estimate. Calibration drifts when a camera is bumped, a lens is refocused or a rig is moved, so a single calibration at the start of a collection campaign may not describe the last episode. Buyers needing metric geometry should expect robot-specific capture, teleoperation or an instrumented rig, not business archives.

Illustrative calibration fields

# Illustrative - synthetic values
calibration_id: cal_site_017_2026-03-04_a
camera_id: wrist_01
intrinsics:
  model: pinhole_radtan   # or another named model, e.g. pinhole_equidistant
  image_size: [1280, 720]
  fx: 0.0   # filled by provider
  fy: 0.0
  cx: 0.0
  cy: 0.0
  distortion: [0.0, 0.0, 0.0, 0.0]
  method: planar_target
  reprojection_rmse_px: 0.0   # buyer sets threshold
extrinsics:
  parent_frame: robot_base
  child_frame: wrist_01
  transform: {translation_m: [0, 0, 0], quaternion_xyzw: [0, 0, 0, 1]}
  handeye_method: "target-based, N poses"
timing:
  camera_to_imu_offset_s: 0.0
valid_from: 2026-03-04T09:12:00Z
valid_until_event: "rig_moved or lens_changed"
report_file: calibration/cal_site_017_2026-03-04_a_report.pdf

Acceptance checks a buyer can run

Pair these with the general acceptance testing for deliveries and the robot trajectory dataset specification. For multi-view recordings of human tasks, multi-view synchronized task recording covers the camera side.

Sourcing notes

Calibration is a property of the collection rig, so it comes from whoever runs the rig: specialist robot-data providers, research labs, or your own team. Requirements to put in a collection brief include recalibration triggers (rig moved, lens or mount changed, start of each session), storage of raw calibration images so you can redo the estimate, and a change log. Rights risk for calibration files themselves is low, but images of calibration targets taken at a workplace can include background content, so check that too. Compare route options in existing assets, new collection or specialist providers.

Common calibration failures to look for

Most calibration defects are visible if you know where to look. Inspect a few frames from each camera at the start, middle and end of a campaign.

• Stale files. One calibration file copied across sessions after a camera was moved. Compare image content, such as a fixed edge or marker, against the calibration's predicted position.
• Mixed lens models. Intrinsics estimated with a pinhole model but applied to a fisheye lens, producing bent straight lines after undistortion.
• Resolution mismatch. Intrinsics estimated at one image size and applied to a resized or cropped stream without scaling.
• Frame confusion. Transforms stored as camera-to-world in one file and world-to-camera in another.
• Unit and order errors. Millimeters versus meters, or quaternion order, silently swapped.
• Auto-focus and zoom. Features that change focal length during an episode invalidate fixed intrinsics.

Ask the provider to state in writing which of these they check, and add a short undistortion check to your own intake.

Limitations and additional preparation

Even well-calibrated data needs preparation: unify conventions across sources, convert formats, and verify that calibration files are tied to the right episodes. If you mix datasets from different rigs, expect to handle different lens models and frames. Calibration accuracy that is adequate for imitation learning may not suit metric tasks such as insertion or 3D reconstruction; the tolerance depends on your task and architecture.

Questions to ask suppliers

1. Which calibration method and tool produced the intrinsics and extrinsics, and can you share the raw output report?
2. How many calibration images or poses were used per camera?
3. How often is the rig recalibrated, and what events trigger it?
4. Is hand-eye calibration target-based or learned, and how is it validated?
5. Are calibration files linked to each episode ID with a validity period?
6. Which frame conventions, units and quaternion order do you use?
7. Do you provide raw calibration images so we can re-estimate?
8. How do you handle lens changes, auto-focus or camera replacement mid-campaign?
9. Are camera-IMU time offsets estimated and stored?
10. Can you show a reprojection check on a held-out target?

Discuss your requirements

If you need calibrated multi-sensor data, describe your rig, sensors and calibration tolerances to SourceX. SourceX looks for US businesses that hold suitable material, checks the data and the supplier's licensing permissions, and coordinates license and delivery. If no existing asset fits, SourceX can scope a commissioned collection of new recordings. A request does not guarantee a match. Start from the humanoid training data guide or the procurement hub.

Discuss your humanoid training-data requirements with SourceX

Sources

• A Flexible New Technique for Camera Calibration — Planar-pattern camera calibration with lens distortion modeling.
• Kalibr — Open toolbox for multi-camera intrinsics/extrinsics and camera-IMU calibration with report outputs.
• DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset — DROID ships camera calibration with each episode.