What Is a Checksum? How S3 Verifies Data Integrity

A checksum is a short, fixed-size value computed from a file's bytes, and changing even one bit of the file changes it. When the checksum you compute before an upload matches the one the server computes on arrival, you know your copy is intact.

TL;DR:

• A checksum proves two copies of a file are identical without comparing every byte. CRC algorithms catch accidental corruption, and SHA-256 also resists deliberate tampering.
• On upload, S3 computes its own checksum and compares it with the one your client sent. A mismatch fails the request with BadDigest.
• The ETag is an MD5 of the data only for single-part uploads that are unencrypted or use SSE-S3. Multipart, SSE-KMS and SSE-C objects get a different ETag.
• On AIOZ Storage, the documented integrity check is rclone check after a migration. Our docs don't describe server-side checksum validation yet.

Updated Oct 19, 2026: answer-first opening, question sections, the rclone check step from our migration guides, and a test script for your own bucket.

What is a checksum?

A checksum is a value computed from data with a fixed algorithm, small enough to compare in an instant. If you compute it before sending a file and again after receiving it, matching values give you strong evidence that nothing changed in between. AWS's documentation puts the purpose plainly: "you can use checksum values to verify the integrity of the data that you upload or download."

Which algorithm you pick depends on what you are protecting against. CRC variants are fast and built to catch accidents, such as a flipped bit on the wire or a bad disk sector. They are not designed to stop someone who crafts a modified file with the same value, which is the job of the SHA family. If your checksum doubles as evidence that nobody tampered with a file, use SHA-256. If you only need to catch transfer errors on large files, a CRC does it with less compute. Content-addressed systems go one step further and make the hash part of the address, which is how IPFS verifies content without trusting the gateway.

How do S3 checksums work?

Two independent calculations have to agree. Per AWS, "all AWS-owned clients calculate a checksum of the object and send it with the upload request," and S3 "independently calculates a checksum value of the object on the server-side" before it stores the object. If the values differ, the request fails with a BadDigest error instead of storing a possibly corrupted file.

S3 supports CRC-32, CRC-32C, CRC-64/NVME, SHA-1, SHA-256, SHA-512, MD5, and three XXHash variants (XXHash64, XXHash3 and XXHash128). CRC-64/NVME is "the default checksum algorithm used for checksum calculations." AWS SDK releases from December 2024 onward calculate a checksum on every upload unless you change the request_checksum_calculation setting from WHEN_SUPPORTED to WHEN_REQUIRED. If you send nothing at all, S3 still computes a CRC-64/NVME value and stores it with the object.

Full object vs composite checksums

A single-part upload gets one checksum over the whole object. Multipart uploads add a choice between two types, which AWS defines this way:

• Full object checksum: calculated over all the content, "from the first byte of the first part to the last byte of the last part."
• Composite checksum: built from the checksums of each part, combined in order from the first part to the last.

Not every algorithm supports both. Per AWS's table for multipart uploads:

AWS explains that only CRC algorithms offer full object checksums "because they can linearize into a full object checksum," so S3 can derive the whole-object value from the part values. Two consequences catch people out. First, CopyObject copies in a single action, so a copy of a composite multipart object gets a new full-object checksum even though the bytes are identical. That mismatch is not corruption. Second, composite checksums need part numbers that are consecutive and start at 1, or completing the upload returns an HTTP 500 error.

Reading a stored checksum back

S3 keeps the checksum as object metadata, and you can ask for it when you read the object:

# checksum of the whole object (must opt in with checksum-mode)
aws s3api head-object --bucket my-bucket --key backup.tar --checksum-mode ENABLED
# checksum plus per-part breakdown for multipart objects
aws s3api get-object-attributes --bucket my-bucket --key backup.tar \
--object-attributes Checksum ObjectParts

GetObjectAttributes is the useful one for multipart objects, because it returns the part count and each part's checksum. You can also fetch one part's checksum by passing a part number to GetObject or HeadObject, or use ListParts while an upload is still open.

For large files, AWS SDKs avoid reading the file twice by sending a trailing checksum. The SDK computes the value while it streams the body with aws-chunked encoding, then appends a header such as x-amz-checksum-crc32 after the last chunk. AWS lists CRC-32, CRC-32C, CRC-64/NVME, SHA-1 and SHA-256 as supported for trailers, and every chunk except the last must be at least 8,192 bytes.

Is the S3 ETag an MD5 hash?

Only sometimes, and this is the most common mistake in S3 integrity checks. Per AWS, the ETag is an MD5 digest of the data only when the object came from PutObject, PostObject or CopyObject and is "plaintext or encrypted by server-side encryption with Amazon S3 managed keys (SSE-S3)." Objects encrypted with SSE-C or SSE-KMS get an ETag that is "not an MD5 digest of its object data," and so does every multipart object, whatever its encryption. The S3 console switches to multipart for files over 16 MB, so the exception covers most large files.

A multipart ETag is built in a documented way. S3 takes the MD5 of each part, concatenates those binary digests, takes the MD5 of the result, and appends a dash and the part count. That is why a three-part upload ends in -3, and why you can rebuild the value locally if you know the part size:

# split into 5 MiB parts, MD5 each part in binary, then MD5 the concatenation
split -b 5242880 -d bigfile part_
for f in part_*; do openssl md5 -binary "$f"; done | md5sum
# ETag = <that hex digest>-<number of parts>

The part size is the catch: the same file uploaded with 8 MiB parts gets a different ETag than with 5 MiB parts. A mismatched multipart ETag can just mean two tools chunked the file differently, so you are better off comparing a stored checksum with a known algorithm.

Verifying data at rest

You can also check data long after it was uploaded. On AWS, the S3 Batch Operations "Compute checksum" operation "allows you to efficiently verify billions of objects in one job request" and writes an integrity report, without downloading or restoring the objects. For a large dataset, that beats downloading everything to confirm nothing degraded.

Outside AWS, the common tool is rclone. rclone check compares sizes and hashes (MD5 or SHA1) between two remotes and lists any files that don't match, without changing either side. When rclone uploads an S3 object as multipart or with SSE-C, where the ETag is no longer an MD5, it writes its own MD5 into an X-Amz-Meta-Md5chksum metadata field and compares that instead. With --download, it downloads both copies and compares the bytes, which works even when a remote has no hashes.

How AIOZ Storage handles checksums

Our migration guides for Amazon S3 and Backblaze B2 end with an integrity step. After rclone sync, the guide says you can "perform a checksum check" and runs:

rclone check --progress s3:your-bucket aioz-storage:your-bucket

The guide calls this the --check flag, but it is the rclone check command shown above. The same guides suggest --ignore-size if a sync fails with "corrupted on transfer: sizes differ." That flag turns off a size comparison, so if you use it, run rclone check afterwards rather than trusting the sync.

To see what the endpoint does with S3 checksum headers, you can run a short test against your own bucket. It sends a SHA-256 checksum on upload and reads the stored value back. We have not run it against AIOZ Storage yet, so the output is yours to record:

import base64
import hashlib
import os
import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    region_name="us-east-1",
    endpoint_url="https://s3.aiozstorage.network",
    aws_access_key_id=os.environ["ACCESS_KEY"],
    aws_secret_access_key=os.environ["SECRET_KEY"],
    config=Config(
        request_checksum_calculation="when_required",
        response_checksum_validation="when_required",
        s3={"addressing_style": "path"},
    ),
)

body = b"checksum test\n" * 1000
sha256 = base64.b64encode(hashlib.sha256(body).digest()).decode()
put = s3.put_object(Bucket="my-test-bucket", Key="sum-test", Body=body,
                    ChecksumSHA256=sha256)
print("PUT ETag:", put["ETag"], "SHA256:", put.get("ChecksumSHA256"))

head = s3.head_object(Bucket="my-test-bucket", Key="sum-test",
                      ChecksumMode="ENABLED")
print("HEAD ETag:", head["ETag"], "SHA256:", head.get("ChecksumSHA256"))

A ChecksumSHA256 value on the read means the checksum was stored. A None means it was not stored or not returned. The client sets request_checksum_calculation to when_required and passes a SHA-256 you computed yourself, so the checksum travels in a header. A checksum the SDK computes for you goes out as an aws-chunked upload with no Content-Length, and AIOZ Storage rejects that, as we confirmed on Oct 19, 2026 (see AWS's data integrity settings). Path-style URLs are needed because bucket subdomains don't resolve. If the upload still fails on the checksum header, drop ChecksumSHA256 and use rclone check for verification instead.

What the docs don't cover yet:

• Whether the endpoint validates x-amz-checksum-* headers and rejects a mismatch with BadDigest.
• Which checksum algorithms are stored and returned.
• Whether the ETag of a single-part upload is an MD5 of the data, given that AIOZ Storage encrypts data server-side.

For the full migration flow that ends with the check, see the AIOZ Storage guide to migrating from Amazon S3.

Frequently Asked Questions

What is a checksum in simple terms?
A short value computed from a file's content that changes if even one bit changes, so you can prove a copy matches the original without comparing every byte.

How does S3 verify that a file was not corrupted during upload?
Your client sends a checksum with the upload, S3 computes its own on arrival, and the upload succeeds only if they match; otherwise it fails with BadDigest.

What checksum algorithm does S3 use by default?
CRC-64/NVME, which S3 computes and stores even when your client sends no checksum at all.

Should I use CRC or SHA-256?
Use a CRC to catch accidental corruption quickly, and SHA-256 when the checksum also has to show that nobody altered the file on purpose.

Why did my object's checksum change after I copied it?
CopyObject copies in one action, so a copy of a multipart object with a composite checksum gets a new full-object checksum even though the data is identical.

How do I check that a migration to AIOZ Storage copied everything correctly?
Run rclone check between the source remote and your AIOZ Storage remote, as the AIOZ migration guides show, and look for files reported as different or missing.

Related Reading

• S3 Multipart Upload Explained, From Part Size to Cleanup - the upload process that changes how the ETag is calculated.
• How AIOZ Storage Encryption Works - the server-side encryption every AIOZ Storage object goes through.
• What Is S3 Object Versioning and How Does It Work? - protection against overwrites rather than corruption.
• Data Durability vs Availability - the wider guarantee checksums help providers deliver.
• SSE-S3 vs SSE-KMS vs SSE-C - the encryption types behind the ETag exceptions.
• Is S3 Eventually Consistent? - when a write becomes visible, as opposed to whether its data is correct.
• S3 Access Logs vs CloudTrail: Which Should You Use? - a record of who accessed an object, not whether its content is intact.

References

• AWS: Checking Object Integrity in Amazon S3
• AWS: Checking Object Integrity for Data Uploads
• AWS SDKs and Tools: Data Integrity Protections for Amazon S3
• rclone: rclone check
• rclone: Amazon S3 Storage Providers (Hashes)
• AIOZ Storage Docs: Migrate Data From Amazon S3
• AIOZ Storage Docs: Migrate Data From Backblaze B2