What Is Multipart Upload? Splitting Files Into Chunks

Multipart upload is how S3-compatible storage handles files too large for a single request: split the file into independent parts, upload each one separately, then ask the server to stitch them back together. A single PutObject call tops out at 5 GB, so anything larger has to go through this process, not as an edge case but as the standard path for any sizeable file. This article covers the exact three-step process, the real size and part-count limits, and what happens to the parts if the upload never finishes.

TL;DR:

• A single S3 upload request maxes out at 5 GB. Multipart upload splits a larger file into parts (5 MiB to 5 GiB each) uploaded independently, then assembled server-side.
• The process is exactly three API calls: initiate, upload each part, complete. Parts can upload in parallel and in any order.
• An abandoned multipart upload keeps billing for its stored parts until it's explicitly completed or aborted, there's no automatic expiry.

Why Large Files Need Multipart Upload at All

A single PutObject request can carry at most 5 GB, per AWS's own S3 documentation, and AWS recommends switching to multipart upload well before that ceiling, at 100 MB, not right at the limit. Splitting a large file into parts and uploading them independently solves three real problems a single giant request doesn't: parts can upload in parallel across multiple connections instead of one slow serial stream, a failed part can be retried on its own instead of restarting the entire upload from byte zero, and an upload can begin before the final file size is even known, useful when a file is being generated on the fly rather than read from disk. None of this is unique to any one vendor, it's how the S3 API itself is built, and any S3-compatible client speaks the same protocol.

The Three-Step Process

Multipart upload is exactly three API calls, run in this order, per AWS's own documentation:

1. Initiate (CreateMultipartUpload): the client requests a new multipart upload for a given object key, and the server responds with an upload ID, a unique identifier that ties every subsequent call for this upload together.
2. Upload parts (UploadPart, one call per part): each part is uploaded with that upload ID and a part number from 1 to 10,000. Parts don't need to be uploaded in order, and AWS says the numbers don't need to be consecutive (1, 5 and 14 is valid), though most clients use them sequentially in practice. Reusing a part number overwrites the earlier part. One exception: when you use part-level (composite) checksums, part numbers must be consecutive and start at 1, or completing the upload returns an HTTP 500. Each part upload returns an ETag that the client has to record for the next step.
3. Complete (CompleteMultipartUpload): the client sends the upload ID plus the full list of part numbers and their ETags. The server concatenates the parts in ascending part-number order into the final object, and only then does the object actually exist and become readable.

Nothing about this requires all parts to come from the same machine or the same network connection. A client can spread parts across multiple parallel workers, which is exactly what makes multipart upload faster for large files on a fast connection, not just more resilient on a flaky one.

The Real Numbers

AWS's own published limits: a part must be between 5 MiB and 5 GiB, except the last part of an upload, which has no minimum size. A single multipart upload can have up to 10,000 parts. Combine those two ceilings and the maximum object size multipart upload supports is 48.8 TiB. AWS's own worked example makes the shape of a real upload concrete: a 100 GB file split into 1,000 parts of 100 MB each requires exactly 1,002 API calls total, one CreateMultipartUpload, 1,000 UploadPart calls, one CompleteMultipartUpload. That's the entire protocol overhead, regardless of file size, since the actual data transfer happens inside the 1,000 part uploads, not as separate bookkeeping.

Multipart upload's parallelism is also why AWS pairs it with S3 Transfer Acceleration in its own documentation: routing each part through the nearest edge location instead of straight to the bucket's home region tackles a different bottleneck than parallel parts alone, network latency over long distances rather than single-connection throughput. AWS's own testing found the combination cutting upload time by up to 61% for large files moved across distance. That's an AWS-specific feature, not a generic S3 API capability every provider ships, worth knowing the difference between the protocol-level benefit (parallelism, works anywhere multipart upload does) and the vendor-specific one (edge routing, works only where that vendor offers it) before assuming both apply outside AWS.

What Happens If You Never Finish

An initiated multipart upload doesn't expire on its own. Every part already uploaded keeps sitting in storage, and keeps being billed for storage, bandwidth, and requests, until the upload is explicitly finished with CompleteMultipartUpload or explicitly cancelled with AbortMultipartUpload. A crashed script, an interrupted process, or a client that just never calls complete leaves those parts billing indefinitely with nothing to show for it. AWS's own guidance for this is a bucket lifecycle rule using the AbortIncompleteMultipartUpload action, which automatically cleans up any multipart upload left unfinished past a set number of days, rather than relying on every client to clean up after itself correctly every time.

What the Three Calls Look Like in the AWS CLI

The high-level commands (aws s3 cp, SDK upload helpers) run all of this for you. The low-level s3api commands expose each step, which is useful for seeing what's actually happening:

# 1. initiate: returns an UploadId
aws s3api create-multipart-upload --bucket my-bucket --key backup.tar

# 2. upload each part with that UploadId (repeat with --part-number 2, 3, ...)
aws s3api upload-part --bucket my-bucket --key backup.tar \
  --part-number 1 --body part-001 --upload-id UPLOAD_ID_FROM_STEP_1

# 3. complete: send the part numbers and the ETag each upload-part returned
aws s3api complete-multipart-upload --bucket my-bucket --key backup.tar \
  --upload-id UPLOAD_ID_FROM_STEP_1 --multipart-upload file://parts.json

where parts.json lists what you recorded:

{
  "Parts": [
    { "PartNumber": 1, "ETag": "\"etag-returned-for-part-1\"" },
    { "PartNumber": 2, "ETag": "\"etag-returned-for-part-2\"" }
  ]
}

AWS is specific about that list: keep your own record of part numbers and ETags, and don't build the complete request from the result of a listing call. The listing is for verification only, and it doesn't include parts that haven't finished uploading. Once the complete request succeeds, AWS notes that "the parts no longer exist," leaving only the assembled object.

Choosing a Part Size

The 10,000-part ceiling sets a floor on part size for big files, and simple division shows how quickly it matters:

That's why a fixed small part size eventually breaks. An 8 MiB part size, the default for boto3's TransferConfig (with an 8 MiB multipart threshold and 10 concurrent threads), can only describe about 78 GiB before it runs out of parts, so tools that upload very large files have to raise the size automatically or let you set it. Check what yours does before a 500 GB upload fails at part 10,001.

Past that floor, the choice is a trade-off. AWS notes a smaller part size "minimizes the impact of restarting a failed upload due to a network error," since a retry re-sends only one part. Larger parts mean fewer requests to make and track. Memory is the other cost: rclone's documentation gives the formula for its S3 backend as extra memory equal to --transfers × --s3-upload-concurrency × --s3-chunk-size. With 4 transfers, 4 concurrent chunks and 16 MiB chunks, for example, that's 256 MiB of buffers, and it scales multiplicatively if you raise any one of them.

In Python, the same knobs live in TransferConfig:

import boto3
from boto3.s3.transfer import TransferConfig

MB = 1024 * 1024
s3 = boto3.client("s3", endpoint_url="https://YOUR-S3-ENDPOINT")

config = TransferConfig(
    multipart_threshold=64 * MB,   # switch to multipart above this size
    multipart_chunksize=64 * MB,   # size of each part
    max_concurrency=8,             # parallel part uploads
)
s3.upload_file("backup.tar", "my-bucket", "backup.tar", Config=config)

Managing Incomplete Uploads

AWS's rule is clear: after you initiate an upload and send at least one part, you must complete or stop it to stop paying for the parts. To find leftovers, list in-progress uploads (up to 1,000 per request, so page through if you have more) and list the parts of a specific one:

aws s3api list-multipart-uploads --bucket my-bucket
aws s3api list-parts --bucket my-bucket --key backup.tar --upload-id UPLOAD_ID
aws s3api abort-multipart-upload --bucket my-bucket --key backup.tar --upload-id UPLOAD_ID

Two details about aborting. After you stop an upload you can't upload any part using that upload ID again. And AWS warns that part uploads already in flight can still succeed or fail after the stop, so to free all the storage you should stop an upload only after all part uploads have finished. Stopping deletes the uploaded parts and there are no early-delete charges for removing incomplete uploads, regardless of storage class.

The set-and-forget fix is the lifecycle rule AWS recommends, which aborts uploads that aren't completed within a set number of days of being initiated:

{
  "Rules": [
    {
      "ID": "abort-stale-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json. AWS states the rule applies to both existing and future multipart uploads, has no effect on completed ones, and doesn't delete any objects. Pick the number of days from your slowest legitimate upload, since a rule shorter than a real upload will abort it mid-flight.

Permissions, Encryption and Concurrent Writers

Every step of a multipart upload is authorized as s3:PutObject on the object. Aborting needs s3:AbortMultipartUpload, listing parts needs s3:ListMultipartUploadParts, and listing in-progress uploads needs s3:ListBucketMultipartUploads on the bucket. If the object is encrypted with SSE-KMS, the requester needs both kms:GenerateDataKey and kms:Decrypt, and AWS notes that without those permissions on CompleteMultipartUpload the object is created without a checksum value. Which key type to choose is covered in SSE-S3 vs SSE-KMS vs SSE-C.

Concurrency is the other surprise. Nothing stops two clients from initiating uploads to the same key. AWS's rule for versioning-enabled buckets is that completing a multipart upload always creates a new version, and the current version is whichever upload was initiated most recently, even if an older one finishes later. Without versioning, another request received between initiation and completion can take precedence, and if a key is deleted mid-upload, the complete call can report success for an object you never see. To stop an upload from replacing something that already exists, add a conditional write to CompleteMultipartUpload, as covered in strong consistency and conditional writes.

One practical ceiling on the other side of the API: AWS lists 160 GB as the maximum file size for uploads through the S3 console, and says anything larger has to go through the CLI, an SDK, or the REST API.

Testing Whether an Endpoint Handles Multipart

Because AIOZ Storage's documentation doesn't address multipart upload, the reliable answer is an experiment. Following the ETag rules from the checksum article, a multipart object's ETag ends in a dash and the part count, so a small forced multipart upload is easy to check:

import os
import boto3
from boto3.s3.transfer import TransferConfig

MB = 1024 * 1024
s3 = boto3.client("s3", endpoint_url="https://YOUR-S3-ENDPOINT")  # credentials via env

with open("test-20mb.bin", "wb") as f:
    f.write(os.urandom(20 * MB))

# 5 MiB parts and a 5 MiB threshold force multipart: expect 4 parts
config = TransferConfig(multipart_threshold=5 * MB, multipart_chunksize=5 * MB)
s3.upload_file("test-20mb.bin", "my-test-bucket", "mpu-test", Config=config)

etag = s3.head_object(Bucket="my-test-bucket", Key="mpu-test")["ETag"]
print(etag)  # a multipart upload ends in -4; a single-part upload has no dash

If the upload succeeds and the ETag ends in -4, the endpoint accepted the three-call protocol. If it errors, or the ETag has no dash because the upload went through as a single request, the endpoint didn't do multipart the way S3 does. The script is a sketch that hasn't been run against AIOZ Storage's endpoint, and a pass on 20 MiB doesn't prove behavior at multi-gigabyte scale.

Does AIOZ Storage Support Multipart Upload?

AIOZ Storage's own documentation doesn't mention multipart upload, chunking, or file splitting anywhere in its concept pages, so treat direct support as unconfirmed rather than assumed either way. If a specific integration depends on it, test against AIOZ Storage's actual S3-compatible endpoint directly before building around it.

Frequently Asked Questions

What's the maximum size for a single S3 upload without multipart?
5 GB per PutObject request. AWS recommends switching to multipart upload starting around 100 MB, well before that hard ceiling.

How many parts can one multipart upload have?
Up to 10,000, numbered 1 through 10,000. Combined with the 5 GiB maximum part size, that puts the largest possible multipart-uploaded object at 48.8 TiB.

What's the minimum size for a part in a multipart upload?
5 MiB, except for the last part of the upload, which has no minimum size at all.

Do the parts of a multipart upload have to be uploaded in order?
No. Part numbers don't need to be consecutive or uploaded sequentially, which is exactly what makes parallel upload across multiple parts possible.

What happens if a multipart upload is never completed?
The uploaded parts keep sitting in storage, and keep being billed, indefinitely. AWS recommends a lifecycle rule using AbortIncompleteMultipartUpload to automatically clean up unfinished uploads after a set number of days.

Is multipart upload only useful for unreliable networks?
No. It also improves throughput on fast, stable connections by uploading parts in parallel, and it lets an upload start before the final object size is known, three separate benefits, not just one.

How do I choose a part size for a multipart upload?
Divide the file size by 10,000 to find the smallest part size that fits, for example about 105 MiB for a 1 TiB file. Above that floor, smaller parts make a failed retry cheaper and larger parts mean fewer requests, and memory use grows with part size times concurrency.

How do I find and delete incomplete multipart uploads?
List them with aws s3api list-multipart-uploads, inspect one with list-parts, and delete it with abort-multipart-upload. To automate cleanup, add a lifecycle rule with AbortIncompleteMultipartUpload and a DaysAfterInitiation value.

Can I reuse an upload ID after aborting a multipart upload?
No. AWS states that after you stop a multipart upload you can't upload any part using that upload ID again, and that in-flight part uploads can still succeed or fail after the stop.

What is the largest file I can upload through the S3 console?
160 GB per AWS's documentation. Larger files need the AWS CLI, an SDK, or the REST API.

What happens if two clients start multipart uploads to the same key?
In a versioning-enabled bucket, each completed upload creates a new version and the current version is the one that was initiated most recently. Without versioning, another request can take precedence between initiation and completion.

Why is a multipart object's ETag different from an MD5 hash?
S3 builds it from the MD5 of each part concatenated and hashed again, with a dash and the part count appended. That's also why a multipart object's ETag isn't an MD5 of the whole file.

Does AIOZ Storage support S3 multipart upload?
Not documented either way. AIOZ Storage's concept pages don't mention chunking or multipart upload, so this isn't confirmed as supported, only untested against the actual endpoint.

Related Reading

• What Is Amazon S3? Object Storage Explained - the bucket/object/key model multipart upload builds objects into.
• Using AIOZ Storage as a Dataset and Model-Output Backend - what AIOZ Storage's own published limits do and don't confirm about large-object handling.
• File Management API with FastAPI, Boto3, and AIOZ Storage - the boto3 client setup any multipart upload code would build on.
• AIOZ Storage Explained: S3-Compatible Decentralized Storage - the full picture of what AIOZ Storage's S3 compatibility does and doesn't cover.
• What Is a Checksum and How S3 Verifies Data Integrity - why a multipart upload changes how the object's ETag actually gets calculated.

References

• AWS Documentation: Uploading and Copying Objects Using Multipart Upload
• AWS Documentation: Amazon S3 Multipart Upload Limits
• AWS Blog: Uploading Large Objects to Amazon S3 Using Multipart Upload and Transfer Acceleration
• AWS: Uploading an Object Using Multipart Upload
• AWS: Configuring a Lifecycle Rule to Delete Incomplete Multipart Uploads
• Boto3: S3 TransferConfig