S3 Multipart Upload Explained, From Part Size to Cleanup

S3 multipart upload sends one large file as up to 10,000 separate parts, then asks the server to join them into one object. You need it for anything over 5 GB, and AWS recommends it from 100 MB.

TL;DR:

• One PutObject request tops out at 5 GB. Multipart upload splits a file into parts of 5 MiB to 5 GiB, with up to 10,000 parts and 48.8 TiB per object.
• It takes three calls: create the upload, upload each part, complete it. Parts can go in parallel and in any order.
• An unfinished upload keeps its parts, and their storage bill, until you complete or abort it. A lifecycle rule cleans them up for you.
• In Python, boto3's TransferConfig switches to multipart on its own once a file passes 8 MiB.

Updated Oct 29, 2026: new sections on multipart checksums and the ETag formula, browser uploads with presigned part URLs, and copying objects over 5 GB, plus a labelled diagram of the three calls.

How does S3 multipart upload work?

Per AWS's documentation, multipart upload is three API calls that you run in this order, each one tied to the same upload ID:

1. Initiate (CreateMultipartUpload): you ask for a new upload to an object key, and the server returns an upload ID that ties every later call together.
2. Upload parts (UploadPart, one call per part): you send each part with that upload ID and a part number from 1 to 10,000, in any order and without needing consecutive numbers. Reusing a number overwrites the earlier part, and each call returns an ETag you must record.
3. Complete (CompleteMultipartUpload): you send the upload ID plus every part number and its ETag. The server joins the parts in ascending order, and only then does the object exist.

One exception to the numbering rule: with part-level (composite) checksums, part numbers must be consecutive and start at 1, or the complete call returns an HTTP 500.

Splitting a file this way solves three problems that a single request can't. Parts upload in parallel over several connections, so a fast link gets used fully, and a failed part is retried on its own instead of restarting from byte zero. You can also start uploading before you know the final size, which helps when a file is generated on the fly. On AWS, Transfer Acceleration adds edge routing for each part, and AWS's own tests cut upload time for large, distant files by up to 61%. Edge routing is an AWS feature, so don't expect it from other S3-compatible providers. Video platforms split large uploads the same way, as in this chunked video upload guide with 50-200 MB parts.

What are the S3 multipart upload limits?

These are AWS's published numbers, and other S3-compatible providers set their own, so check yours before you size a job:

The protocol overhead stays small even for big files. AWS's worked example splits a 100 GB file into 1,000 parts of 100 MB, which takes 1,002 calls: one create, 1,000 part uploads, and one complete, with the data itself travelling inside the part uploads.

Choosing a part size

Start from the 10,000-part ceiling: divide your file size by 10,000 and you get the smallest part size that fits.

This is why a fixed small part size eventually breaks: boto3's default of 8 MiB parts can describe only about 78 GiB before it runs out of part numbers. If you upload bigger files, raise the part size yourself or check that your tool raises it for you.

Above that floor, you are trading cheap retries against request count. AWS notes that a smaller part size "minimizes the impact of restarting a failed upload due to a network error," because a retry re-sends one part, while larger parts mean fewer requests to send and track.

Memory is the third cost, and rclone's docs give the extra memory for its S3 backend as --transfers × --s3-upload-concurrency × --s3-chunk-size. With 4 transfers, 4 concurrent chunks and 16 MiB chunks, you hold 256 MiB of buffers. By default, rclone switches to multipart at 200 MiB and sends 5 MiB chunks, 4 at a time per file.

The three calls in the AWS CLI

High-level commands such as aws s3 cp run all of this for you, while the low-level s3api commands show each step on its own:

# 1. initiate: returns an UploadId
aws s3api create-multipart-upload --bucket my-bucket --key backup.tar

# 2. upload each part with that UploadId (repeat with --part-number 2, 3, ...)
aws s3api upload-part --bucket my-bucket --key backup.tar \
  --part-number 1 --body part-001 --upload-id UPLOAD_ID_FROM_STEP_1

# 3. complete: send the part numbers and the ETag each upload-part returned
aws s3api complete-multipart-upload --bucket my-bucket --key backup.tar \
  --upload-id UPLOAD_ID_FROM_STEP_1 --multipart-upload file://parts.json

Here parts.json lists the part numbers and ETags you recorded during step 2:

{
  "Parts": [
    { "PartNumber": 1, "ETag": "\"etag-returned-for-part-1\"" },
    { "PartNumber": 2, "ETag": "\"etag-returned-for-part-2\"" }
  ]
}

AWS is specific here: keep your own record of part numbers and ETags, and don't build the complete request from a listing call. The listing is for checking only, and it leaves out parts that are still in flight. Once the complete call succeeds, AWS says "the parts no longer exist," and only the joined object remains.

Checksums and the multipart ETag

S3 can check a multipart upload in two ways. A full-object checksum covers every byte from the first part to the last, while a composite checksum is built from the checksums of the separate parts.

The algorithm you pick decides which type you get. CRC64NVME, CRC32 and CRC32C support full-object checksums, because CRC values from separate parts can be combined into one value for the whole file. SHA-1, SHA-256 and MD5 work only as composite checksums on a multipart upload. If you send no checksum at all, AWS says S3 attaches a full-object CRC64NVME value for you. When you pass a full-object value in the complete call and it doesn't match, S3 fails the request with a BadDigest error.

The ETag follows a separate rule. S3 takes the MD5 of each part, joins those digests, hashes the result, and appends a dash and the part count. You can work out the expected ETag before you upload, as long as you use the same part size as your client:

import hashlib

def multipart_etag(path, part_size=8 * 1024 * 1024):
    """Expected S3 ETag for a file uploaded in parts of part_size bytes."""
    digests = []
    with open(path, "rb") as f:
        while chunk := f.read(part_size):
            digests.append(hashlib.md5(chunk).digest())
    combined = hashlib.md5(b"".join(digests)).hexdigest()
    return f'"{combined}-{len(digests)}"'

We ran this function here on a 20 MiB file of zero bytes with 5 MiB parts, and it printed "0cdaffc8e69cab00470bf84bc31f305b-4". It hasn't been compared with a live S3 response, so treat it as a check of the formula AWS documents. Change the part size and the ETag changes too, even for an identical file, which makes ETags a poor way to compare two copies. The checksum and data integrity guide covers the algorithms in more depth.

Uploading parts from a browser with presigned URLs

You can let a browser upload a large file straight to the bucket without handing it your secret key. AWS's Compute Blog describes the pattern: your backend starts the upload, presigns a URL for each part, and completes the upload once the parts are in.

1. The backend calls CreateMultipartUpload and keeps the upload ID.
2. It presigns one UploadPart URL per part. Each URL carries its own part number and the upload ID, so a 40-part file needs 40 URLs.
3. The browser sends each part with a PUT to its URL and records the ETag from each response.
4. The browser returns the part numbers and ETags, and the backend calls CompleteMultipartUpload.

Two details trip people up here. Per MDN, a browser script can read only the CORS-safelisted response headers unless the server lists others in Access-Control-Expose-Headers, and ETag isn't on that list. Your bucket's CORS rule has to expose ETag, or step 3 has nothing to record (see how CORS works for object storage). The other detail is lifetime: on AWS, a URL presigned with an IAM user's keys lasts at most 7 days under Signature Version 4. Signing short-lived URLs in code is covered in presigned URLs with Python.

Copying an object larger than 5 GB

Multipart upload also handles copies. A single CopyObject request copies at most 5 GB, so AWS says a larger object has to be copied as a multipart upload. Each part then uses UploadPartCopy, which names the source object in the x-amz-copy-source header instead of carrying data.

To copy one slice of the source into a part, add x-amz-copy-source-range: bytes=first-last with zero-based byte offsets, so bytes=0-9 copies the first 10 bytes. Ranges work only when the source is larger than 5 MB. Because the server reads from the source object, you don't download and re-upload the data yourself. AWS adds that a copy keeps the speed benefit of a multipart object only when you make it with the CLI or an SDK, not the S3 console.

What happens to an unfinished multipart upload?

It stays open, and you keep paying for the stored parts until you complete or abort it. AWS bills an open upload for its storage, bandwidth and requests, at the rate of the storage class you named for the parts. A crashed script or a client that never calls complete leaves those parts behind with nothing to show for them, so it pays to know how to find and remove leftovers:

aws s3api list-multipart-uploads --bucket my-bucket
aws s3api list-parts --bucket my-bucket --key backup.tar --upload-id UPLOAD_ID
aws s3api abort-multipart-upload --bucket my-bucket --key backup.tar --upload-id UPLOAD_ID

After an abort, you can't upload a part with that upload ID again afterwards. Part uploads already in flight can still succeed after the abort, so AWS advises aborting only once they have finished. Aborting deletes the parts, with no early-delete charge in any storage class.

The set-and-forget fix is the lifecycle rule AWS recommends, which aborts any upload that isn't completed within a set number of days:

{
  "Rules": [
    {
      "ID": "abort-stale-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json. AWS says the rule covers existing and future uploads, leaves completed ones alone, and deletes no objects. Set the day count longer than your slowest real upload, or the rule will abort that upload mid-flight.

Permissions, encryption and two writers on one key

Every step of a multipart upload is authorized as s3:PutObject on the object. Aborting needs s3:AbortMultipartUpload, listing parts needs s3:ListMultipartUploadParts, and listing open uploads needs s3:ListBucketMultipartUploads. With SSE-KMS, you also need kms:GenerateDataKey and kms:Decrypt. Without them on the complete call, AWS says the object is created without a checksum value. The key types are compared in SSE-S3 vs SSE-KMS vs SSE-C.

Nothing stops two clients from starting uploads to the same key at the same time. In a versioned bucket, each completed upload becomes a new version, and the current one is whichever was initiated last. Without versioning, another write that lands between start and finish can win. To stop an upload from replacing an existing object, add a conditional write to the complete call, as covered in strong consistency and conditional writes.

How AIOZ Storage handles multipart upload

AIOZ Storage speaks the S3 API at https://s3.aiozstorage.network. Our Python SDK guide sets up boto3 with region us-east-1 and path-style addressing, so TransferConfig is where you control part size and concurrency. The docs have no multipart page yet, so the script below is a test to run on your own bucket. It has not been run against AIOZ Storage.

import os
import boto3
from boto3.s3.transfer import TransferConfig
from botocore.config import Config

MB = 1024 * 1024
s3 = boto3.client(
    "s3",
    region_name="us-east-1",
    endpoint_url="https://s3.aiozstorage.network",
    aws_access_key_id=os.environ["ACCESS_KEY"],
    aws_secret_access_key=os.environ["SECRET_KEY"],
    config=Config(
        request_checksum_calculation="when_required",
        response_checksum_validation="when_required",
        s3={"addressing_style": "path"},
    ),
)

with open("test-20mb.bin", "wb") as f:
    f.write(os.urandom(20 * MB))

# 5 MiB parts and a 5 MiB threshold force multipart: expect 4 parts
config = TransferConfig(
    multipart_threshold=5 * MB,
    multipart_chunksize=5 * MB,
    max_concurrency=4,
)
s3.upload_file("test-20mb.bin", "my-test-bucket", "mpu-test", Config=config)

etag = s3.head_object(Bucket="my-test-bucket", Key="mpu-test")["ETag"]
print(etag)  # multipart: ends in -4; single request: no dash

The two when_required settings are there because AIOZ Storage needs a Content-Length header on uploads. Since 1.36.0, boto3 otherwise sends uploads chunked without one (see AWS's data integrity settings and the botocore Config reference). Path-style URLs are required too, because bucket subdomains don't resolve. The CLI calls earlier need the same two changes on AIOZ Storage: export AWS_REQUEST_CHECKSUM_CALCULATION=when_required and aws configure set default.s3.addressing_style path (AWS CLI S3 configuration). We confirmed both on Oct 19, 2026.

If the upload succeeds and the ETag ends in -4, the endpoint ran the three-call protocol. If it errors, or the ETag has no dash, the upload did not go through as multipart. A pass at 20 MiB says nothing about multi-gigabyte files, so repeat the test at the size you really plan to upload. The checksum guide explains why a multipart ETag carries that dash.

Keep the request limit in mind when you set concurrency, because our docs list a default of 100 requests per second and each part is its own request. At 8 MiB parts on a 1 Gbit/s link, you send roughly 15 part requests per second, well under that ceiling.

What the docs don't cover yet:

• Whether multipart upload and its list and abort calls are supported.
• Whether each part upload counts toward the 100 requests per second limit.
• Whether lifecycle rules such as AbortIncompleteMultipartUpload work, and how stored parts of an open upload are billed.
• The maximum object and part size.

To get keys and a bucket ready first, follow the AIOZ Storage Python SDK guide.

Frequently Asked Questions

What is the largest file I can upload to S3 without multipart?
5 GB per PutObject request, although AWS recommends switching to multipart upload from about 100 MB.

How many parts can one multipart upload have?
Up to 10,000, which with 5 GiB parts puts the largest object you can build at 48.8 TiB on Amazon S3.

What is the minimum part size?
Every part must be at least 5 MiB, except the last part of the upload, which has no minimum size.

Do parts have to be uploaded in order?
No, you can upload them in any order and in parallel, and the server joins them by part number when you complete the upload.

How do I find and delete incomplete multipart uploads?
List them with aws s3api list-multipart-uploads and delete each with abort-multipart-upload, or add a lifecycle rule with AbortIncompleteMultipartUpload to clean them up automatically.

Why is a multipart object's ETag not an MD5 of the file?
S3 hashes the MD5 of each part together and appends a dash and the part count, so the ETag can't be compared with an MD5 of the whole file.

Does AIOZ Storage support S3 multipart upload?
Our docs don't say either way yet, so run the 20 MiB test above against your own bucket before you build a workflow around it.

Related Reading

• Object storage glossary - short definitions of multipart upload, lifecycle rule and the other terms used here.
• What Is Amazon S3? Object Storage Explained - the bucket, object and key model multipart upload builds into.
• Using AIOZ Storage as a Dataset and Model-Output Backend - what AIOZ Storage's published limits do and don't confirm about large objects.
• File Management API with FastAPI, Boto3, and AIOZ Storage - the boto3 client setup this code builds on.
• AIOZ Storage Explained: S3-Compatible Decentralized Storage - what AIOZ Storage's S3 compatibility covers.
• What Is a Checksum and How S3 Verifies Data Integrity - how multipart upload changes an object's ETag.
• AIOZ Storage Rate Limit: Working Within 100 Requests/s - the documented 100 requests per second default.

References

• AWS Documentation: Uploading and Copying Objects Using Multipart Upload
• AWS Documentation: Amazon S3 Multipart Upload Limits
• AWS Documentation: Uploading Objects
• AWS: Uploading an Object Using Multipart Upload
• AWS: Configuring a Lifecycle Rule to Delete Incomplete Multipart Uploads
• Boto3: S3 TransferConfig
• rclone: Amazon S3 Storage Providers
• AIOZ Storage Docs: Usage Limit
• AWS: Checking Object Integrity for Data Uploads in Amazon S3
• AWS API Reference: UploadPartCopy
• AWS: Copying, Moving, and Renaming Objects
• AWS Compute Blog: Uploading Large Objects to Amazon S3 Using Multipart Upload and Transfer Acceleration
• AWS: Download and Upload Objects with Presigned URLs
• MDN: Access-Control-Expose-Headers