SFTP for Dataset Delivery: Setup, Keys and Limits at Scale

This page sits in the schemas, packaging and delivery hub and covers the SFTP channel only. Bucket-based handover is covered in cross-account cloud bucket delivery, and file-level encryption in encrypting dataset deliveries with PGP, age and KMS.

Why enterprise suppliers still ask for SFTP

Suppliers prefer SFTP because it is the transfer protocol their security and IT teams have already approved. A finance operations group exporting ledger extracts, or a support organization exporting ticket histories from Zendesk or Salesforce Service Cloud, often has a managed file transfer tool (GoAnywhere, Cerberus, MOVEit, IBM Sterling) and a runbook for pushing files to partner SFTP endpoints. Asking that team to assume an AWS IAM role or mint an Azure SAS token can add weeks of internal review.

SFTP also gives you a clean boundary for acceptance. The supplier pushes, you pull nothing from their network, and every file lands in an account you control and audit. That matters when the license defines which records you received and when, because your SFTP logs and manifest checks become the delivery record. For background on how licensed data typically moves, see how licensed data is delivered.

Key-based authentication, chroot and network controls

Accept only SSH public keys, give each supplier its own account jailed to its own directory, and allow nothing but SFTP. On a self-hosted OpenSSH server, the standard pattern is a Match Group block with ChrootDirectory and ForceCommand internal-sftp, with AllowTcpForwarding no and X11Forwarding no. The chroot path must be owned by root and not writable by the user, so create a supplier-writable incoming/ subdirectory inside it.

Two failure modes recur in practice. First, the default AuthorizedKeysFile sits in the user's home directory, so a supplier with write access there could add or replace its own keys; point it to a root-owned path such as /etc/ssh/authorized_keys/%u so key control stays with you. Second, teams leave PasswordAuthentication yes enabled globally for one legacy partner, which reopens brute-force exposure for every supplier account.

Network controls complete the setup:

• IP allow-list: restrict port 22 (or your chosen port) to the supplier's published egress ranges in a security group or firewall rule, and record the ranges in the delivery specification.
• Key hygiene: require Ed25519 or RSA keys of at least 3072 bits, one key pair per supplier system (not per person), and a rotation date in the delivery spec.
• Logging: set LogLevel VERBOSE so the key fingerprint used for each login is recorded, and ship the logs to your SIEM.
• No reuse: never share an account between two suppliers, even affiliated ones; the account boundary is how you prove whose files arrived.

These are transport controls. They encrypt the channel and authenticate the sender, but they do not protect files at rest on your landing volume, which is why many buyers also require file-level encryption.

Managed SFTP endpoints backed by object storage

Managed SFTP services remove the server-patching burden and write straight into object storage, which makes the later move to bucket-based pipelines easier. AWS Transfer Family is the common example (per its documentation as of October 2026): with service-managed users, each user holds one or more SSH public keys stored in the service, and login succeeds only when the client's private key matches. Each user gets a home directory mapped to an S3 prefix, and the restricted option keeps the user inside that folder, with a session policy available for finer control.

One detail affects design: a Transfer Family server supports a single authentication method, chosen when the server is created. If one supplier insists on passwords, you need a custom identity provider or a separate server, not a relaxation of the main one. Azure Blob Storage and Google Cloud offer comparable SFTP front ends or partner products; check each service's current documentation for key types, quotas and per-session limits as of October 2026 before you commit a supplier to it.

The practical benefit is that your landing zone is already an S3 or Blob prefix. Event notifications can trigger the verification job the moment a manifest file arrives, without a cron job polling a Linux filesystem.

Layering PGP on top of SFTP

Use OpenPGP file encryption when files will sit at rest on a shared landing volume, pass through a supplier's managed file transfer hub, or need a signature proving which supplier key produced them. SFTP protects bytes in transit only. OpenPGP, now specified in RFC 9580, adds a session-key-encrypted payload and optional signatures that travel with the file [1].

The common pattern: you publish an encryption public key, the supplier signs with its own key and encrypts to yours, and your quarantine job verifies the signature before decrypting. Keep the decryption key in a KMS or HSM-backed service, not on the SFTP host. Agree on file naming (.pgp or .gpg), ASCII armor versus binary, and compression in the delivery spec so a mismatch does not stall acceptance. Key exchange and rotation details are covered in the dataset encryption and key exchange guide, and transit questions in whether licensed data is encrypted in transit.

Quarantine landing zone and manifest verification

Treat every SFTP upload as untrusted until a manifest check passes, and promote files to the training data lake only after that. A supplier upload can be partial, interrupted mid-file, duplicated after a retry, or simply the wrong export. The landing zone design below prevents a half-written Parquet file or truncated JSONL shard from reaching a training job.

A workable layout uses three areas: incoming/ (supplier-writable), quarantine/ (system-only), and accepted/ (read by pipelines). The supplier uploads data files first and the manifest last, or uploads to a temporary name and renames on completion; your job starts only when the manifest appears. Packaging conventions such as BagIt, which pairs a payload directory with checksum manifests and tag files, give suppliers a ready-made format for this [2]. Deeper guidance is on the manifest and checksum verification page.

Illustrative example: invented to show structure; it does not describe an available dataset.

Step 5 and 6 tie the channel to content. Link them to your CSV delivery specification and stable record IDs so a recurring delivery can be joined to the last one.

Throughput and resume limits at terabyte scale

SFTP's main weakness is throughput: a single session runs over one TCP connection with windowed request acknowledgment, so long-distance transfers rarely fill a fast link, and large files make every interruption expensive. Typical clients (OpenSSH sftp, WinSCP, lftp) can resume a partial upload with reput or equivalent, but resume relies on byte offsets and has no built-in integrity check, so a resumed file still needs its checksum verified.

Workarounds exist but add operational load. Suppliers can split exports into many shards of 1 to 10 GB and upload in parallel sessions, which helps throughput and limits rework on failure. Expect friction with managed file transfer tools that cap concurrent sessions, and with appliances that inspect SSH traffic.

Signs you have outgrown SFTP:

• A single delivery takes longer than the agreed delivery window, even with parallel shards.
• Re-uploads after interruptions are routine rather than rare.
• You are running custom scripts to fan uploads across sessions.
• The data already lives in the supplier's cloud object storage, so SFTP adds a download-then-upload hop.

SFTP vs S3 and other channels for large files

Choose SFTP for supplier convenience and moderate volumes, object storage for scale and parallelism, and offline transfer for datasets too large for the network. Cross-account bucket delivery uses a bucket policy in the owner account and delegated user policies in the other account [3]; Azure uses scoped, time-limited SAS tokens, preferably user delegation SAS [4]. Both support multipart, parallel uploads with per-part integrity checks that SFTP lacks.

Illustrative example: invented to show structure; it does not describe an available dataset.

For one-off very large transfers, see offline transfer appliances. For recurring supply, the incremental versus full refresh guide helps you size each drop, which often decides the channel. Put the chosen channel, endpoints, key fingerprints and verification steps in a technical delivery specification.

How SourceX handles delivery

SourceX sources operational datasets from US companies on request and manages the commercial process through licensing and ongoing purchases. Delivery runs through private, access-controlled workflows, never email attachments, and only after an executed agreement and supplier approval. Every dataset is rights-reviewed and delivered under a license that defines the records, uses, term and delivery. Personal details are removed or replaced before delivery, the method is recorded and a sample is checked, though no method is perfect. Teams evaluating suppliers can describe the data they need on the SourceX buyers page.

Sourcing supplier datasets for your delivery pipeline

SourceX sources operational datasets from US companies on request and runs the process from Find and Assess through Agree, Transact and Manage, with nothing contracted until a supplier agrees. Delivery happens through private, access-controlled workflows after an executed agreement. Describe the data you need at sourcex.si/buyers.