How to find and export your Amazon S3 data for AI training

01

What is Amazon S3 data, and why do AI labs and data vendors buy it?

What Amazon S3 stores

Amazon S3 (Simple Storage Service) is a file storage service run by Amazon Web Services (AWS). Files are called objects, and they are kept in buckets, which are named containers. An object's name, called its key, often works like a folder path, so a key such as inspections/2026/04/job-1042/roof.jpg says which job a photo belongs to.

Many other systems write into S3 without anyone noticing: a website saves uploaded photos there, a phone system saves call recordings, and a database saves its backups and exports. A bucket can therefore hold years of files from several programs side by side.

Why files tied to records teach an AI model something

An AI model learns by studying a very large number of examples and finding the patterns in them, and those examples are called training data. A photo alone shows only what a roof looks like. A photo whose key names a job, which a database record ties to a diagnosis and a repair, shows what the roof looked like and what the expert concluded about it.

AI labs now want to train AI agents, software that carries out a job step by step, and an agent learns a job from records of that job being done. Our guide AI training data: why AI companies want your company's data explains the mechanism.

How Amazon S3 fits into a dataset

An S3 bucket is usually one of several sources in a company's dataset, next to email, documents and tickets. Companies are paid $100K to $2M or more per dataset, according to micro1, the data vendor we typically refer companies to. The price covers the whole dataset, not each source on its own, and what your data is worth explains what moves it.

02

Where does Amazon S3 show how much data you have?

How to read your total in the console

An AWS administrator (the person who manages the company's AWS account) does these steps. No paid tier is needed, but the person must have the right permissions. The dashboard steps follow Amazon's guide to viewing Storage Lens dashboards.

1. Sign in with an IAM user or role (an AWS login that has been granted specific permissions) that may view S3 Storage Lens and read CloudWatch. The account's root user, the original login created with the AWS account, cannot open Storage Lens dashboards.
2. In the S3 console's left navigation, go to Storage Lens > Dashboards and pick default-account-dashboard.
3. On the Overview tab, the Snapshot section shows Total storage and Object count. The Buckets and Storage classes tabs break the totals down.
4. To look at one bucket, open it and go to its Metrics tab, which charts daily storage under Bucket metrics. CloudWatch holds identical figures in its AWS/S3 namespace.

How to read one bucket from the command line

CloudWatch stores one size figure per bucket per storage class each day, so the command asks for one class at a time. This example reads a day of storage for the Standard class, and you repeat it for each other StorageType you use, such as StandardIAStorage or GlacierStorage:

aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes --dimensions Name=BucketName,Value=my-bucket Name=StorageType,Value=StandardStorage --start-time 2026-09-10T00:00:00Z --end-time 2026-09-11T00:00:00Z --period 86400 --statistics Average

Get object-level detail with an inventory report

An S3 Inventory report is a file listing every object in a bucket with the fields you ask for. Open the bucket's Management tab and select Create inventory configuration in the Inventory configurations area. Choose between Include all versions and Current versions only, then tick the fields Size, Last modified date and Storage class. The first report can take up to 48 hours to arrive.

What each figure counts

Why the number can be larger than the files you see

BucketSizeBytes adds up the size of every object and its metadata, both current and noncurrent versions, and the parts of any incomplete multipart uploads, according to Amazon's CloudWatch metrics reference. A bucket with versioning turned on, which keeps every earlier version of a changed or deleted file, can therefore report far more than the files you can see. The free Storage Lens metrics cover the account and its buckets, while paid advanced metrics go down to the prefix level and keep 15 months of history.

03

What should you know about your Amazon S3 data before you apply?

The size in GB

Report the total storage in GB, and check whether versioning is on. If it is, the metric includes old versions, so build an inventory report limited to Current versions only and add up its Size column to get the live figure.

The storage classes

Three groups of objects are archived and cannot be read or copied until they are restored: those in S3 Glacier Flexible Retrieval, those in S3 Glacier Deep Archive, and those in the Archive Access and Deep Archive Access tiers of S3 Intelligent-Tiering. Restoring adds time and cost to an export, so note how much of your data is in those classes. Objects in S3 Glacier Instant Retrieval need no restore.

The years of history

The oldest objects show up when you sort an inventory report by last modified date, and the bucket's creation date is an alternative starting point. Last modified is the later of an object's creation and its last change, so files that were rewritten look newer than they are.

The number of seats

For file storage, the seat count is the number of users, devices or customers whose files the buckets hold. The AWS logins that manage the buckets are not seats.

What you don't need to do first

Applying shares no data with anyone, and you don't need to export or clean anything first. The data vendor anonymizes records before any AI lab sees them, and nothing moves until a lawyer-reviewed license is signed. The application and each later step are laid out in how to sell your company's data.

04

How do you export your full Amazon S3 history?

Step 1: list what you will copy

Create an S3 Inventory report, as described above, for every bucket you intend to copy. The report is a checklist of how many objects there are, how large they are and where they are stored, and it can also be the manifest, the list of objects, that a Batch Operations job reads.

Step 2: restore archived objects

Amazon's guide to restoring archived objects covers both ways to do it. Only objects in the archived classes need this step.

1. For a few objects, open the bucket in the S3 console, select the objects, choose Actions > Initiate restore, set how many days the restored copy stays available and pick a retrieval tier.
2. For many objects, create an S3 Batch Operations job with the Restore operation, give it your inventory as the manifest and set the same number of days. One job handles either the Glacier classes or the Intelligent-Tiering archive tiers, not both, so restoring both kinds needs two jobs.

A restore from S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive makes a temporary copy, so AWS bills both the archived original and the restored copy until the copy expires. Intelligent-Tiering restores at the Standard or Bulk tier have no retrieval charge and return the object to the Frequent Access tier.

Step 3: copy the bucket

The AWS command-line tool copies a whole bucket to local disk or to another bucket, and it copies only objects that are not there yet if you rerun it:

aws s3 sync s3://my-bucket /data/my-bucket
aws s3 sync s3://my-bucket s3://destination-bucket

The tool skips Glacier objects and prints a warning. After their restores finish, run the same command again with --force-glacier-transfer added.

Step 4: use Batch Operations for very large or cross-account copies

1. Start an S3 Batch Operations job that uses the Copy operation, with your inventory as the manifest, and create it in the destination Region, meaning the Region you are copying into.
2. On a versioned bucket where order matters, copy the noncurrent versions in one job, wait for it to finish, then copy the current versions in a second job, because objects are not necessarily copied in manifest order.
3. Compare the destination's object count and total size with the inventory report.

Amazon documents these limits for the Copy operation in its Batch Operations copy guide: objects up to 5 GB each, manifests of up to 20 billion objects, all source objects in one bucket and all destination objects in one bucket. If the destination bucket is unversioned, an object with the same key is overwritten. Use aws s3 sync for objects larger than 5 GB.

What the count and the copy include

The figures S3 reports and the files a sync copies are not the same set. The metric counts old versions and unfinished uploads, while a sync copies only the current version of each object and skips archived ones until they are restored.

Keep version history with Batch Operations

The aws s3 sync command takes only the latest version of every object. To keep the older versions too, build an inventory that lists all versions and have a Batch Operations job copy them.

How long an export takes

Amazon publishes no time for a copy, because it depends on the number and size of the objects, the Regions involved and any archived objects waiting for restore. The first inventory report alone can take up to 48 hours, so start that step early.

Sources: Amazon S3 User Guide, Viewing S3 Storage Lens metrics on the dashboards, CloudWatch metrics and dimensions, Restoring an archived object and Copy objects with Batch Operations. Steps checked against these pages in November 2026.

05

What stays private when Amazon S3 data is licensed?

You choose what is in scope

A copy can be limited to the buckets and the prefixes (the folder-like beginnings of object keys) you name, and a bucket that is not copied never leaves your account. Because keys often carry customer names or job numbers, review them before you decide what is in scope. The data vendor deletes files too sensitive to clean safely instead of anonymizing them.

Names are replaced before a lab sees them

Anonymization means replacing every detail that identifies a person or a company, such as a name or an email address, with a stand-in that is the same every time it appears. Files raise a harder case than database fields, because faces in photos, voices in recordings and names printed inside documents are identifying details too. Our guide to how company data is anonymized for AI training explains how each kind is handled, what is deleted outright and how you review samples.

When files are matched to database records, such as inspection photos to work orders, the match is kept through the stand-ins. The photo still points to the same job, and the job no longer names the customer.

Ownership and payment

Licensing gives the buyer limited rights to a copy, while the company keeps ownership of the originals. Files you created are generally yours to license, but files that belong to customers or other companies may not be, and a lawyer's review settles that, as covered in whether it is legal to sell company data. The terms that matter, such as exclusivity, permitted use and deletion, are covered in data licensing. micro1 pays in full within 30 days of accepting the data, usually as a single payment.