AIOZ Storage as a Dataset and Model-Output Backend

Using AIOZ Storage as a dataset and model-output backend means treating it as exactly what it is: an S3-compatible object store, not a purpose-built MLOps platform. That distinction decides what you should and shouldn't expect from it. This article covers what actually works today, training data in and model outputs out, using the buckets, presigned URLs, and access grants already documented, and where the honest boundaries sit: no native object versioning, no lifecycle policies, and nothing dataset-format-aware built in.

TL;DR:

• AIOZ Storage can reliably hold training datasets and model outputs as plain objects, with presigned URLs and access grants for controlling who reads and writes them.
• It has no documented object versioning or lifecycle/tiering policies, features purpose-built ML storage platforms often ship natively.
• Where you land depends on the tradeoff: lower cost and a familiar S3 API, versus features you'd have to build yourself.

Updated Oct 19, 2026: corrected a line that said AIOZ Storage has no vector database. The docs now document a Vector Database.

What "Dataset and Model-Output Backend" Actually Means Here

Two directions, both handled the same way at the storage layer. Datasets in: training data, images, or any input a job needs to read before it runs. Outputs out: checkpoints, generated files, logs, anything a job produces that needs to persist after it finishes. AIOZ Storage doesn't distinguish between the two, both are just objects in a bucket, which is exactly the point. It is a storage layer, not a training or orchestration layer, and this article stays scoped to what happens at that layer.

What AIOZ Storage Can Do

Upload and download at whatever scale your job needs. AIOZ's published usage limits document 100 buckets and 100 sub-users per account and a 100 request-per-second rate limit, but name no maximum object size or bucket size, so a multi-gigabyte checkpoint file isn't documented as a problem the way it would be against a service with an explicit per-object cap. Organize datasets and outputs by bucket and key prefix, the same structure every S3-compatible tool already expects, and reachable through the JavaScript, Go, Python, and CLI paths already covered on this hub.

Two access-control tools matter specifically for a pipeline, not just a person. Presigned URLs work against AIOZ Storage exactly like they work against S3, since signing happens client-side and doesn't depend on anything AIOZ-specific: hand a training job a time-limited download link for a dataset, or a time-limited upload link for its output, without ever giving it a standing credential. For access that needs to persist longer than a single job run, an access grant scoped to Read on the dataset bucket and Write on the output bucket does the same job with an expiration you control, up to a year out.

What AIOZ Storage Can't (or Doesn't Yet) Do

No documented object versioning. AIOZ's public documentation does not describe a versioning feature, the ability to keep prior copies of an object as it gets overwritten, the way S3 buckets can be configured to do. If a pipeline overwrites latest-checkpoint.pt in place, the previous version is gone, and there is no built-in undo. The practical workaround, covered below, is naming discipline rather than a platform feature.

No documented lifecycle or tiering policies. There is no published mechanism for auto-expiring old objects, auto-deleting stale experiment outputs, or moving cold data to a cheaper storage class, all common S3 lifecycle-rule features. Anything like that has to be scripted and run on a schedule yourself, there's no server-side rule to configure.

No dataset-format awareness. AIOZ Storage stores whatever bytes you give it. It doesn't understand Parquet, WebDataset, or any ML-specific packaging format any more than raw S3 does. Vector search is a separate feature: the AIOZ Storage Vector Database stores and queries embeddings through the AWS S3 Vectors SDKs, but you create the embeddings with your own model, and it doesn't read your dataset files for you. Format-aware processing still happens in a layer your pipeline builds on top.

A Practical Pattern: Structuring Buckets Without Native Versioning

Since there's no built-in version history, the workaround is putting the version in the key instead of relying on the platform to remember it. A dataset bucket organized as datasets/<name>/<version>/... and an output bucket organized as outputs/<run-id>/<checkpoint-name> means every write is a new key, not an overwrite, and nothing gets silently lost. This is the same pattern teams use against plain S3 when they choose not to enable versioning for cost reasons, it isn't an AIOZ-specific workaround, just the standard answer to the same standard gap.

If manual naming discipline isn't enough, DVC (Data Version Control), an open-source tool built for exactly this problem, layers real versioning on top of any S3-compatible remote, tracking data and model versions through Git commits while the actual bytes sit in your bucket. Its dvc remote modify <name> endpointurl <url> command is documented specifically for pointing DVC at non-AWS S3-compatible services like MinIO, the same config shape AIOZ Storage's own endpoint fits, so an AIOZ Storage bucket is a legitimate DVC remote target, the same way it's a legitimate target for any other S3-compatible tool covered on this hub.

Pair that with the access-control split above: a read-scoped access grant or presigned URL for the dataset bucket a training job pulls from, and a separate write-scoped one for the output bucket it writes to. Two different scopes for two different directions of traffic, rather than one broad credential that can do both, keeps a misbehaving job from being able to overwrite the dataset it's supposed to only be reading.

Where This Fits Against Purpose-Built ML Storage Platforms

Worth being direct about the tradeoff rather than dodging it. Platforms built specifically for ML artifacts, dataset registries and experiment-tracking tools with built-in versioning, lineage, and metadata search, do things AIOZ Storage doesn't attempt to do at the storage layer. What AIOZ Storage offers instead is a plain, S3-compatible object store on decentralized infrastructure, at the pricing and encryption model covered elsewhere on this hub. If your pipeline already treats S3 as a dumb, reliable object layer and handles versioning or metadata itself, that's a straightforward swap. If it depends on a platform's native versioning or lineage features, those don't come with the endpoint, and building them yourself is real, non-trivial work worth planning for upfront.

Frequently Asked Questions

Can I store large model checkpoint files on AIOZ Storage?
AIOZ's published usage limits document account-level caps (100 buckets, 100 sub-users, 100 requests per second) but no maximum object or bucket size, so a large checkpoint file isn't documented as a problem the way it would be against a service with an explicit per-object cap.

Does AIOZ Storage support object versioning?
Not that its public documentation describes. There is no documented feature for keeping prior versions of an overwritten object. The workaround is either a version identifier in the object key itself, or a tool built for this specifically, like DVC, which layers real versioning on top of any S3-compatible remote through its endpointurl config option.

Can I set up automatic expiration for old training outputs?
Not through a documented AIOZ Storage feature. There's no published lifecycle-policy mechanism for auto-expiring or auto-tiering objects; cleanup has to be scripted and scheduled yourself.

How do I give a training job temporary access to a dataset without sharing my main credentials?
A presigned URL, generated client-side with the same boto3 method used against Amazon S3, or a time-scoped access grant if the job needs longer-lived access than a single presigned link's expiration window.

Does AIOZ Storage understand ML-specific data formats like Parquet or WebDataset?
No. It stores objects as bytes, the same as any S3-compatible store, with no built-in awareness of ML packaging formats.

Is AIOZ Storage a replacement for a dedicated ML dataset-versioning platform?
Not on its own. It handles the storage layer reliably, but versioning, lineage, and metadata search that a dedicated dataset-registry tool provides natively would need to be built on top, not assumed to come with the endpoint.

Related Reading

• AIOZ Storage for AI and Data Pipelines: What's Actually Documented Today - the hub this article is part of, with real deployment examples.
• Generating Presigned URLs on AIOZ Storage with Python - the exact mechanism for handing a job temporary, scoped access to a bucket.
• AIOZ Storage Access Control: Macaroons and Access Grants - scoping read/write access by bucket for longer-lived pipeline credentials.
• File Management API with FastAPI, Boto3, and AIOZ Storage - the Python client setup this article's examples build on.
• AIOZ Storage Explained: S3-Compatible Decentralized Storage - the full pricing and account-limits picture referenced above.
• What Is Retrieval-Augmented Generation (RAG)? Explained - a common consumer of the model outputs and datasets this backend stores, grounding an LLM's answers in retrieved data.
• What Is Object Lock and How Does S3 WORM Storage Work? - the industry feature this article's versioning gap directly implies AIOZ Storage doesn't support.
• What Is S3 Object Versioning and How Does It Work? - the mechanism itself, delete markers, version IDs, and the real storage cost, covered in full.
• Offloading LangGraph Agent Checkpoints to AIOZ Storage - a real, working AI workload storing structured checkpoint data rather than raw datasets or model output.

References

• AIOZ Storage Documentation: Usage Limits
• AWS S3 User Guide: Managing the Lifecycle of Objects
• DVC Documentation: Amazon S3 and S3-Compatible Remote Storage