What Is a Checksum and How S3 Verifies Data Integrity

What is a checksum, and why does object storage lean on one so heavily for something as basic as confirming a file arrived intact? AWS's own documentation states the core idea plainly: "you can use checksum values to verify the integrity of the data that you upload or download." A checksum is a short, fixed-size value calculated from a file's actual bytes, change even one bit of the file and the checksum changes with it, which makes it a reliable way to prove two copies of a file are genuinely identical without comparing every byte directly. This article covers how S3 actually validates one on upload, which algorithms it supports, and a specific gotcha that trips up a lot of people who assume the ETag is always a simple MD5 hash.

TL;DR:

• A checksum is a value calculated from a file's content that changes if even one bit of the file changes, used to prove a copy is identical to the original without comparing every byte.
• On upload, S3 independently calculates its own checksum and compares it to the one the client sent, per AWS's own documentation, rejecting the upload if they don't match.
• The ETag is often assumed to always be an MD5 digest, and AWS's own documentation confirms that's only true for single-part, unencrypted or SSE-S3-encrypted uploads, not for multipart uploads or objects using SSE-C/SSE-KMS encryption.

What Is a Checksum?

A checksum is a fixed-size value computed from a piece of data using a specific algorithm, small enough to compare quickly, but sensitive enough that changing even a single byte of the original data produces a completely different checksum. That property is what makes it useful for integrity verification: if you calculate a checksum before sending a file and the checksum calculated after receiving it matches exactly, you have strong evidence nothing was altered or corrupted along the way, without needing to compare the entire file byte-for-byte.

How S3 Actually Verifies Integrity on Upload

AWS's own documentation describes the actual mechanism, not just the concept: "for uploads, all AWS-owned clients calculate a checksum of the object and send it with the upload request. S3 then independently calculates a checksum value of the object on the server-side, and validates it with the provided value before storing the object and checksum value." Two independent calculations, one from the client before the data leaves, one from S3 after it arrives, have to agree before the object is accepted at all. If they don't match, AWS states the request fails with a BadDigest error rather than silently storing a possibly-corrupted file.

The S3 Checksum Algorithms Supported

S3 doesn't limit you to one algorithm. AWS's documentation lists a full set of supported options: CRC-32, CRC-32C, CRC-64/NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants (XXHash64, XXHash3, XXHash128). CRC-64/NVME is notable because AWS states it's "the default checksum algorithm used for checksum calculations," meaning every object gets some form of integrity protection even if the uploader never thinks about it directly.

The default works in two layers. If you use an upgraded AWS SDK and don't pass an algorithm, the SDK picks one and calculates the checksum for you. If neither you nor the SDK calculates anything, S3 falls back to CRC-64/NVME on its own and attaches it to the object. Either way, you can override the choice by setting ChecksumAlgorithm on the request.

Which one to pick depends on what you're protecting against. CRC variants are fast and built to catch accidental corruption, such as a flipped bit on the wire or a bad disk sector. They aren't designed to resist someone deliberately crafting a modified file with the same value, and that's the case the SHA family is built for. If the checksum doubles as evidence that a file wasn't tampered with, SHA-256 is the safer default. If the goal is only to catch transfer errors on large files quickly, a CRC does the job with less compute.

Full Object vs Composite Checksums in S3

For a single-part upload, the checksum covers the whole object and that's the end of it. Multipart uploads add a choice. AWS defines two checksum types:

• Full object checksum: calculated over all the content, "from the first byte of the first part to the last byte of the last part."
• Composite checksum: calculated from the individual checksums of each part. Instead of hashing the data again, S3 aggregates the part-level checksums, first part to last, into one combined value.

Not every algorithm supports both. Per AWS's table for multipart uploads:

The split follows from how the math works. AWS explains that full object checksums are only available for CRC-based algorithms "because they can linearize into a full object checksum," which lets S3 compute the whole-object value from the part-level values and parallelize the work. SHA and MD5 can't do that, so they're composite only.

Two consequences catch people out:

1. Copying a multipart object can change its checksum. A CopyObject call does the copy as a single action, so the result carries a direct checksum of the full object. If the original was a composite multipart checksum, the value changes even though the data is byte-for-byte identical. A checksum mismatch after a copy isn't automatically corruption.
2. Part numbers must be consecutive, starting at 1 when you use composite checksums. AWS states that completing an upload with non-consecutive part numbers returns an HTTP 500 Internal Server error.

Reading a Stored Checksum Back

S3 stores the checksum value as object metadata alongside the object, and you can request it at download time. Two calls cover most cases:

# checksum of the whole object (must opt in with checksum-mode)
aws s3api head-object --bucket my-bucket --key backup.tar --checksum-mode ENABLED

# checksum plus per-part breakdown for multipart objects
aws s3api get-object-attributes --bucket my-bucket --key backup.tar \
  --object-attributes Checksum ObjectParts

GetObjectAttributes is the useful one for multipart objects. It returns how many parts make up the object and, with additional checksums enabled, each part's checksum value. AWS also notes you can fetch a single part's checksum from a completed upload by passing a part number to GetObject or HeadObject, and use ListParts for uploads still in progress.

Trailing Checksums: Verifying in a Single Pass

For large uploads, computing the checksum before sending means reading the file twice, once to hash it and once to send it. AWS SDKs avoid that with trailing checksums. When you set ChecksumAlgorithm, the SDK calculates the checksum as it streams the data and appends it to the end of a chunked upload request, so S3 verifies and stores in one pass.

Mechanically, the request uses aws-chunked content encoding. The client sends an x-amz-trailer header naming the checksum header that will follow the body, such as x-amz-checksum-crc32, and an x-amz-content-sha256 value of STREAMING-UNSIGNED-PAYLOAD-TRAILER (or STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER when SigV4 payload signing is used). Body chunks must be at least 8,192 bytes except the last one. AWS lists five algorithms as supported for trailers: CRC-32, CRC-32C, CRC-64/NVME, SHA-1 and SHA-256. You'll rarely build this by hand, but it explains why a well-behaved SDK upload gets integrity checking without a second read of the file.

The ETag Trap: Why It's Not Always an MD5 Digest

This is the part worth remembering above everything else in this article, since it's the single most common mistake around S3 integrity checking. Plenty of tooling and documentation casually treats the ETag as "the MD5 hash of the file," and AWS's own documentation is specific about exactly when that's actually true and when it isn't. An object gets an ETag that's a genuine MD5 digest of its data only "if an object is created by the PutObject, PostObject, or CopyObject operation... and that object is also plaintext or encrypted by server-side encryption with Amazon S3 managed keys (SSE-S3)."

Two common cases break that assumption entirely. AWS states directly that an object "encrypted by server-side encryption with customer-provided keys (SSE-C) or server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS)" gets an ETag that is "not an MD5 digest of its object data." And separately, any object "created by either the multipart upload process... [has] an ETag [that] is not an MD5 digest, regardless of the method of encryption," which matters more than it might seem, since AWS's own console automatically uses multipart upload for any object larger than 16 MB. A script comparing a locally-computed MD5 hash against the ETag of a large or KMS-encrypted object isn't validating anything, it's comparing two values that were never supposed to match in the first place.

How the Multipart ETag Is Actually Built

The FAQ-level answer is "a multipart ETag isn't an MD5 digest." The mechanism is more specific, and AWS documents it: S3 calculates an MD5 of each part as it's uploaded, concatenates the bytes of those digests, calculates an MD5 of the concatenated result, and adds a dash followed by the total number of parts. That's why a multipart ETag ends in a suffix like -3 for a three-part upload.

That structure means you can reproduce it locally if you know the part size the uploader used:

# split into 5 MiB parts, MD5 each part in binary, then MD5 the concatenation
split -b 5242880 -d bigfile part_
for f in part_*; do openssl md5 -binary "$f"; done | md5sum
# ETag = <that hex digest>-<number of parts>

The catch is the part size. The same file uploaded with 8 MiB parts produces a different ETag than one uploaded with 5 MiB parts, because the per-part digests differ. So a mismatched multipart ETag can simply mean the two uploads chunked the file differently. A stored checksum with a known algorithm is a far more reliable comparison than reverse-engineering an ETag.

Verifying Data at Rest, Not Just on Upload

Integrity checking isn't limited to the moment of upload. AWS's documentation describes a batch operation specifically for checking data already sitting in a bucket: the "Compute checksum" operation, run through S3 Batch Operations, which "allows you to efficiently verify billions of objects in one job request" and produces "an automatically generated integrity report... that you can use to confirm that your data set remains intact," all without downloading or restoring the objects being checked. For a large existing dataset, that's a meaningfully different and more practical approach than trying to re-download everything just to confirm nothing has silently degraded.

Where AIOZ Storage Fits

AIOZ Storage's own documentation doesn't describe a checksum feature, a supported algorithm list, or an ETag integrity mechanism anywhere, confirmed directly against its concept-definitions page rather than assumed. Since AIOZ Storage is S3-compatible, standard S3 clients and SDKs may still send or expect certain checksum-related headers as part of normal request behavior, but whether AIOZ Storage's own backend independently validates them the way S3 does, or simply stores an object without that server-side check, isn't documented either way. That's worth testing directly for any workload where integrity verification genuinely matters, rather than assuming S3-compatibility implies an identical integrity-checking implementation underneath.

Frequently Asked Questions

What is a checksum in simple terms?
A short value calculated from a file's content that changes if even one bit of the file changes, used to prove a copy is identical to the original without comparing every byte directly.

How does S3 verify a file wasn't corrupted during upload?
The client calculates a checksum before sending the file, S3 independently calculates its own checksum after receiving it, and the upload is only accepted if the two values match. A mismatch fails the request with a BadDigest error.

What checksum algorithms does S3 support?
CRC-32, CRC-32C, CRC-64/NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants. CRC-64/NVME is the default algorithm applied automatically when no other one is specified.

Is the S3 ETag always an MD5 hash of the file?
No, and assuming so is a common mistake. It's only a true MD5 digest for single-part uploads that are unencrypted or use SSE-S3 encryption. Multipart uploads and objects encrypted with SSE-C or SSE-KMS get an ETag that isn't an MD5 digest at all.

How can you verify the integrity of objects already stored in a bucket, not just new uploads?
With S3's "Compute checksum" batch operation, which can verify billions of existing objects in one job and produce an integrity report, without downloading or restoring any of the data being checked.

Does AIOZ Storage support checksum verification?
Not documented. AIOZ Storage's own documentation doesn't describe a checksum feature or algorithm list, confirmed directly against its concept pages, so whether its backend independently validates integrity the way S3 does isn't confirmed either way.

What is the difference between a full object and a composite S3 checksum?
A full object checksum is calculated over all of an object's content, first byte to last. A composite checksum is calculated from the checksums of each individual part of a multipart upload. CRC-based algorithms support both types for multipart uploads, while SHA, MD5 and XXHash algorithms support composite only.

Why did an object's checksum change after I copied it in S3?
CopyObject copies the object as a single action, so the copy gets a direct checksum of the full object. If the original was a multipart upload with a composite checksum, the value changes even though the data is identical.

How do I see an object's checksum with the AWS CLI?
Run aws s3api head-object with --checksum-mode ENABLED for the object's checksum, or aws s3api get-object-attributes with --object-attributes Checksum ObjectParts to also see the part count and each part's checksum on multipart objects.

Should I use CRC or SHA-256 for S3 checksums?
CRC variants are fast and suited to catching accidental corruption in transit. They aren't designed to resist deliberate tampering, so use SHA-256 when the checksum also has to serve as evidence a file wasn't altered.

Why does multipart upload change how the ETag works?
Because a multipart-uploaded object's ETag is calculated from the individual parts' checksums combined, not a single hash of the whole file's bytes, per AWS's own documentation. That makes it structurally different from a simple MD5 digest, regardless of encryption.

Related Reading

• What Is Multipart Upload? How S3 Splits Large Files Into Chunks - the upload process that directly changes how the ETag is calculated, covered here.
• How AIOZ Storage Encryption Actually Works - the encryption modes referenced in the ETag/SSE-S3 distinction in this article.
• What Is S3 Object Versioning and How Does It Work? - another integrity-adjacent mechanism, protecting against overwrites rather than corruption.
• Data Durability vs Availability: What 11 Nines Means - the broader guarantee checksums help providers actually deliver on.
• SSE-S3 vs SSE-KMS vs SSE-C: S3 Encryption Types Explained - the exact encryption types behind this article's ETag-isn't-always-MD5 caveat.
• Eventual vs Strong Consistency: How S3 Made the Switch - a different guarantee, when a write becomes visible, distinct from whether the data itself is correct.
• S3 Access Logs vs CloudTrail: Which Should You Use? - a different kind of record entirely, who accessed an object, rather than whether its content is intact.

References

• AWS: Checking Object Integrity in Amazon S3
• AWS: Checking Object Integrity for Data Uploads