What Is a Checksum and How S3 Verifies Data Integrity

What is a checksum, and why does object storage lean on one so heavily for something as basic as confirming a file arrived intact? AWS's own documentation states the core idea plainly: "you can use checksum values to verify the integrity of the data that you upload or download." A checksum is a short, fixed-size value calculated from a file's actual bytes, change even one bit of the file and the checksum changes with it, which makes it a reliable way to prove two copies of a file are genuinely identical without comparing every byte directly. This article covers how S3 actually validates one on upload, which algorithms it supports, and a specific gotcha that trips up a lot of people who assume the ETag is always a simple MD5 hash.

TL;DR:

• A checksum is a value calculated from a file's content that changes if even one bit of the file changes, used to prove a copy is identical to the original without comparing every byte.
• On upload, S3 independently calculates its own checksum and compares it to the one the client sent, per AWS's own documentation, rejecting the upload if they don't match.
• The ETag is often assumed to always be an MD5 digest, and AWS's own documentation confirms that's only true for single-part, unencrypted or SSE-S3-encrypted uploads, not for multipart uploads or objects using SSE-C/SSE-KMS encryption.

What Is a Checksum?

A checksum is a fixed-size value computed from a piece of data using a specific algorithm, small enough to compare quickly, but sensitive enough that changing even a single byte of the original data produces a completely different checksum. That property is what makes it useful for integrity verification: if you calculate a checksum before sending a file and the checksum calculated after receiving it matches exactly, you have strong evidence nothing was altered or corrupted along the way, without needing to compare the entire file byte-for-byte.

How S3 Actually Verifies Integrity on Upload

AWS's own documentation describes the actual mechanism, not just the concept: "for uploads, all AWS-owned clients calculate a checksum of the object and send it with the upload request. S3 then independently calculates a checksum value of the object on the server-side, and validates it with the provided value before storing the object and checksum value." Two independent calculations, one from the client before the data leaves, one from S3 after it arrives, have to agree before the object is accepted at all. If they don't match, AWS states the request fails with a BadDigest error rather than silently storing a possibly-corrupted file.

The Checksum Algorithms S3 Supports

S3 doesn't limit you to one algorithm. AWS's documentation lists a full set of supported options: CRC-32, CRC-32C, CRC-64/NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants. CRC-64/NVME is notable specifically because AWS states it's "the default checksum algorithm used for checksum calculations" when no other algorithm is specified, meaning every object gets some form of integrity protection even if the uploader never thinks about it directly.

The ETag Trap: Why It's Not Always an MD5 Digest

This is the part worth remembering above everything else in this article, since it's the single most common mistake around S3 integrity checking. Plenty of tooling and documentation casually treats the ETag as "the MD5 hash of the file," and AWS's own documentation is specific about exactly when that's actually true and when it isn't. An object gets an ETag that's a genuine MD5 digest of its data only "if an object is created by the PutObject, PostObject, or CopyObject operation... and that object is also plaintext or encrypted by server-side encryption with Amazon S3 managed keys (SSE-S3)."

Two common cases break that assumption entirely. AWS states directly that an object "encrypted by server-side encryption with customer-provided keys (SSE-C) or server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS)" gets an ETag that is "not an MD5 digest of its object data." And separately, any object "created by either the multipart upload process... [has] an ETag [that] is not an MD5 digest, regardless of the method of encryption," which matters more than it might seem, since AWS's own console automatically uses multipart upload for any object larger than 16 MB. A script comparing a locally-computed MD5 hash against the ETag of a large or KMS-encrypted object isn't validating anything, it's comparing two values that were never supposed to match in the first place.

Verifying Data at Rest, Not Just on Upload

Integrity checking isn't limited to the moment of upload. AWS's documentation describes a batch operation specifically for checking data already sitting in a bucket: the "Compute checksum" operation, run through S3 Batch Operations, which "allows you to efficiently verify billions of objects in one job request" and produces "an automatically generated integrity report... that you can use to confirm that your data set remains intact," all without downloading or restoring the objects being checked. For a large existing dataset, that's a meaningfully different and more practical approach than trying to re-download everything just to confirm nothing has silently degraded.

Where AIOZ Storage Fits

AIOZ Storage's own documentation doesn't describe a checksum feature, a supported algorithm list, or an ETag integrity mechanism anywhere, confirmed directly against its concept-definitions page rather than assumed. Since AIOZ Storage is S3-compatible, standard S3 clients and SDKs may still send or expect certain checksum-related headers as part of normal request behavior, but whether AIOZ Storage's own backend independently validates them the way S3 does, or simply stores an object without that server-side check, isn't documented either way. That's worth testing directly for any workload where integrity verification genuinely matters, rather than assuming S3-compatibility implies an identical integrity-checking implementation underneath.

Frequently Asked Questions

What is a checksum in simple terms?
A short value calculated from a file's content that changes if even one bit of the file changes, used to prove a copy is identical to the original without comparing every byte directly.

How does S3 verify a file wasn't corrupted during upload?
The client calculates a checksum before sending the file, S3 independently calculates its own checksum after receiving it, and the upload is only accepted if the two values match. A mismatch fails the request with a BadDigest error.

What checksum algorithms does S3 support?
CRC-32, CRC-32C, CRC-64/NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants. CRC-64/NVME is the default algorithm applied automatically when no other one is specified.

Is the S3 ETag always an MD5 hash of the file?
No, and assuming so is a common mistake. It's only a true MD5 digest for single-part uploads that are unencrypted or use SSE-S3 encryption. Multipart uploads and objects encrypted with SSE-C or SSE-KMS get an ETag that isn't an MD5 digest at all.

How can you verify the integrity of objects already stored in a bucket, not just new uploads?
With S3's "Compute checksum" batch operation, which can verify billions of existing objects in one job and produce an integrity report, without downloading or restoring any of the data being checked.

Does AIOZ Storage support checksum verification?
Not documented. AIOZ Storage's own documentation doesn't describe a checksum feature or algorithm list, confirmed directly against its concept pages, so whether its backend independently validates integrity the way S3 does isn't confirmed either way.

Why does multipart upload change how the ETag works?
Because a multipart-uploaded object's ETag is calculated from the individual parts' checksums combined, not a single hash of the whole file's bytes, per AWS's own documentation. That makes it structurally different from a simple MD5 digest, regardless of encryption.

Related Reading

• What Is Multipart Upload? How S3 Splits Large Files Into Chunks - the upload process that directly changes how the ETag is calculated, covered here.
• How AIOZ Storage Encryption Actually Works - the encryption modes referenced in the ETag/SSE-S3 distinction in this article.
• What Is S3 Object Versioning and How Does It Work? - another integrity-adjacent mechanism, protecting against overwrites rather than corruption.
• Data Durability vs Availability: What 11 Nines Means - the broader guarantee checksums help providers actually deliver on.
• SSE-S3 vs SSE-KMS vs SSE-C: S3 Encryption Types Explained - the exact encryption types behind this article's ETag-isn't-always-MD5 caveat.
• Eventual vs Strong Consistency: How S3 Made the Switch - a different guarantee, when a write becomes visible, distinct from whether the data itself is correct.
• S3 Access Logs vs CloudTrail: Which Should You Use? - a different kind of record entirely, who accessed an object, rather than whether its content is intact.

References

• AWS: Checking Object Integrity in Amazon S3
• AWS: Checking Object Integrity for Data Uploads