What nobody tells you about storage migrations

Copying objects from one bucket to another is a solved problem. The command is one line and it has been one line for a decade.

Migrations still go wrong constantly, and they go wrong in the same nine ways every time. None of them are about the copy. All of them are about everything around the copy.

Here is the list, in roughly the order they bite.

1. You do not actually know what is in the bucket

Almost every migration starts with a number that turns out to be wrong.

You think you have 4 TB. You have 4 TB of current objects, plus 2 TB of noncurrent versions because versioning has been on since 2023, plus an unknown quantity of incomplete multipart uploads that do not appear in a normal listing but are being billed.

Do this first: get object count, total size, size distribution, version count, and incomplete multipart upload count. aws s3api list-multipart-uploads is the one people have never run.

The size distribution matters more than the total. Object count drives migration time far more than total bytes. Five million small files is a much harder problem than five thousand large ones, and the tool that handles one well may handle the other badly.

2. Archived objects cannot be copied

If any of your data is in Glacier Flexible Retrieval or Deep Archive, a copy operation will fail on those objects. They have to be restored to a retrievable state first.

Restore takes minutes to hours for Flexible, and up to twelve hours for Deep Archive. It also costs a retrieval fee.

This is the single longest lead time in any migration and it is the one most often discovered on the day. Start the restore first, before anything else, ideally days ahead.

3. ETags lie about multipart objects

The most common source of false failure reports.

For a single-part upload, the ETag is the MD5 of the object, so comparing ETags between source and destination is a reasonable integrity check.

For a multipart upload it is not an MD5. It is a hash of the concatenated part hashes with the part count appended, like -14. It depends on the part size used at upload time.

So when you copy a multipart object to a new provider with a different part size, the ETags will not match, and nothing is wrong.

If your verification compares ETags blindly, you will see a wall of mismatches on exactly your largest and most important objects, and you will either panic or, worse, stop trusting the verification entirely.

What to do: compare object count and size for everything, and use ETag comparison only below the multipart threshold. For genuine content verification on large objects, compute your own checksums on both sides.

4. Storage classes do not map cleanly

S3 has around seven storage classes. Most destinations have one or two.

Copy from Glacier Instant Retrieval to a provider with a single class and everything lands in that class. That is usually what you want and occasionally very much not. Objects designed to be cheap-and-slow become normal-price-and-fast, and your bill reflects it.

Check the class distribution of your source before you start, and decide deliberately what each class maps to.

5. Request limits, not bandwidth, are the constraint

People plan migrations around bandwidth. Almost nobody plans around request rate.

Every object is at least one LIST entry, one GET, and one PUT. Millions of objects means millions of requests, and both source and destination will throttle you.

Symptoms: the transfer starts fast and progressively slows, or you see 503s and rate-limit errors. The fix is backoff and lower concurrency, not more parallelism, which is the instinct and makes it worse.

Also: requests cost money. Class A and Class B operation charges on a few million objects are small but not zero, and they are absent from every migration cost estimate I have ever seen.

6. Metadata quietly does not come across

Object bytes copy reliably. Everything attached to them is less certain.

• Content-Type usually copies, and when it does not, browsers download files instead of displaying them. This is the most user-visible migration failure and it is invisible in every storage-level check.
• Custom metadata (x-amz-meta-*) usually copies. Verify.
• Object tags frequently do not, and are frequently what your lifecycle rules select on.
• ACLs often have no equivalent at the destination.
• Storage class, as above.

Test through your application, not just the storage API. A migration can be byte-perfect and still break your site because a Content-Type header changed.

7. Nobody defines what "done" means

The question that should be answered before the transfer starts and almost never is: how will we know this worked?

Without a definition agreed in advance, verification becomes a judgement call made at 1am by a tired person who wants it to be finished. That person always decides it is finished.

Write it down first. For example: object counts match, total sizes match, 100 randomly sampled objects byte-match, the application reads correctly from the new endpoint for 48 hours, and error rates are unchanged.

8. The cutover is the risky part, not the copy

The copy is a background job. The cutover is a moment where production changes.

The failure mode is straightforward: you copy on Monday, cut over on Friday, and everything written between Monday and Friday is missing from the destination.

The pattern that works:

1. Bulk copy everything
2. Run a delta sync to catch what changed
3. Cut over reads to the new endpoint, keep writes going to both if you can
4. Run for a period with both live
5. Final delta sync
6. Cut over writes
7. Wait a week before deleting the source

That last step is the one people skip to save a month of storage, and it is the most expensive saving available.

9. The transfer has a price tag, sometimes two

Your source provider charges you to send data out. Moving 10 TB out of S3 costs roughly $911 in AWS egress. That is unavoidable and it is billed to you by AWS.

What is avoidable is a second charge on top from the tool. Several migration tools bill per gigabyte:

• Movebot: $0.75/GB, so 10 TB costs $7,500
• Flexify: $0.02 to $0.04 per GiB, so 10 TB is roughly $200 to $400
• MultCloud: traffic-metered, with $9.90 a month covering 100 GB
• Cloudsfer: subscription with a cap, plus overage up to $0.75/GB

Check what the transfer itself costs before you start, because it is often larger than the storage saving for the first year.

There are also free paths worth knowing about regardless of what tool you use. If your destination is Cloudflare R2, their own Super Slurper imports from S3 on Cloudflare's infrastructure at no transfer charge. If you are moving into R2 and paying anyone per gigabyte, you are paying for something that exists for free.

The checklist

Before

• Object count, total size, size distribution
• Version count and incomplete multipart uploads
• Storage class distribution
• Restore anything archived (start this first)
• Confirm the destination supports what you need (object lock, versioning, classes)
• Price the transfer: source egress plus tool charges
• Write down what "done" means
• Test on one prefix, end to end, including reading through the application

During

• Monitor for throttling; reduce concurrency rather than increasing it
• Confirm resume works by interrupting it deliberately
• Log what completed

After

• Object count matches
• Total size matches
• Sample byte-level verification, aware of the multipart ETag caveat
• Content-Type spot-check
• Application reads correctly from the new endpoint
• Delta sync for anything written during the copy
• Both live in parallel for a period
• Wait a week before deleting the source

The honest bit

Most migrations go fine. This article is long because the minority that go badly go badly in predictable ways, not because migration is inherently dangerous. If you have 500 GB of images in one storage class with no versioning, you can ignore most of this and run rclone copy.

The hard part is not technical, it is organisational. The step people skip is verification, and they skip it because it is boring and because by the time the copy finishes everyone wants to be done. Every migration horror story I have heard reduces to somebody declaring success without checking.

Tools do not remove the need to think. We make one. It handles resume, reports what completed, and does not charge per gigabyte. It cannot tell you whether your application depends on an ACL you are about to lose. Nothing can, except you, in advance.

Questions people actually ask

How long does a 10 TB migration take? Hours to days. Object count matters more than size. A million small objects can take longer than the same bytes in a thousand large ones.

Can I migrate without downtime? Usually yes, with the bulk-copy, delta-sync, parallel-run pattern. Downtime is normally a choice made to simplify the cutover, not a requirement.

What is the most common migration failure? Incomplete verification. The copy usually works. The check that would have caught the one thing that did not is usually the step that got skipped.

Should I use my provider's own migration tool? If it exists and it is free, usually yes, at least for the bulk phase. Cloudflare's Super Slurper is a good example. They are typically one-directional and one-time, so you may still need something else for ongoing sync.

Do I need to migrate everything at once? No, and you should not. Prefix by prefix is slower and dramatically safer, and it lets you learn on data that matters less.

How do I verify a 5 million object migration? Counts and sizes for everything, statistical sampling for content. Byte-comparing 5 million objects costs more in requests than the migration did.

---

Storafleet runs migrations server-side with resume, reports what completed, and does not charge per gigabyte. Both providers connect into one console, so the delta sync during cutover is a scheduled job rather than a script you maintain. Connect a bucket, free, no card.