Incremental sync and cutover patterns with rclone and S3 APIs

She is three hours into a cutover. The old bucket is read-only, the new bucket is live, and the diff report says 412 objects exist in one place and not the other. An rclone sync is running in a terminal on the jump host. A spreadsheet of ETags is open on the second monitor. She's about to discover that multipart ETags are not checksums.

We have heard this scene described in detail, often, by engineers who lived it. Not one specific person, but the same shape every time: a migration that was going fine until the moment it had to become a cutover.

rclone sync is not a cutover tool, and the S3 API will not tell you that

Incremental sync is the easy half, and it is genuinely easy. rclone sync is one of the best-documented commands in open-source infrastructure. You point it at two remotes, you give it --checksum if you do not trust size-and-mtime, you point --backup-dir somewhere safe so overwritten and deleted files survive, you run it once, then you run it again an hour later and it moves almost nothing. That second run is the whole promise of incremental sync: bytes on the wire drop to near zero because the tool only ships what changed.

Cutover is a different operation wearing the same clothes.

Cutover means freezing writes, redirecting clients, and then proving the two sides agree. The freeze is not a flag. It is a conversation with every team that has a writer pointed at that bucket, and a DNS or config change, and a rollback plan if the diff comes back wrong. rclone has no opinion about any of that. Neither does the S3 API. There is no --cutover switch, and there is no endpoint that returns a reconciliation report a human can sign off on.

What you get instead is a tool that tells you what it would copy. That is not the same as telling you the destination is correct.

We built Storafleet partly because of that gap, but we are not going to pretend the gap is closed by software alone. It is closed by freezing writes and comparing both sides after the freeze. Everything else is bookkeeping.

We use the S3 API's own listing semantics against it, because that is where drift hides

A second rclone sync after cutover can still miss objects, and the reasons are boring and specific. None of them are rclone bugs. They are properties of S3-compatible storage.

Start with pagination. ListObjectsV2 returns at most 1000 keys per response and hands you a ContinuationToken. If your reconciliation script takes the first page and stops, or mishandles the token, you are comparing a sample and calling it a diff. This is the single most common way a "clean" migration report is wrong.

curl -s "https://s3.example.com/?list-type=2&prefix=logs/&max-keys=1000" \
  -H "Authorization: AWS4-HMAC-SHA256 ..." \
  | jq -r '.Contents[] | [.Key, .ETag, .Size, .LastModified] | @tsv' \
  > page1.tsv

# then follow NextContinuationToken until IsTruncated is false
curl -s "https://s3.example.com/?list-type=2&prefix=logs/&continuation-token=$TOKEN" \
  -H "Authorization: AWS4-HMAC-SHA256 ..." \
  | jq -r '.Contents[] | [.Key, .ETag, .Size, .LastModified] | @tsv' \
  >> page1.tsv

Next, ETags. Multipart ETags are not checksums and they are not comparable across providers. An object uploaded in five parts has an ETag that looks like "abc123...-5". The hash inside it is an MD5 over the concatenated part MD5s, not over the object. Change the part size and the ETag changes for identical bytes. Upload through a provider that computes things differently and the ETag is different again. Any diff that compares ETags between two providers and reports mismatches is reporting on part sizes, not on data.

If you need real comparison, use the checksum fields. ChecksumSHA256 and friends are returned by providers that support them, and rclone check with --checksum will do a proper job when both ends expose one. When they do not, you are back to downloading and hashing, and you should budget for that instead of assuming it will be free.

Then there is LastModified clock skew. It is server-set, it is second-resolution, and "the source object is newer, so copy it" falls apart when the two providers' clocks disagree by seconds or when a write lands inside the same second as the listing. rclone's default size-and-modtime comparison is right most of the time. Most of the time is not a sign-off criterion.

The practical version of this is a three-step reconciliation. Freeze writes. List both sides with full pagination and store key, size, and whatever checksum the provider actually gives you. Compare on key and size, flag every key that differs, and treat ETag as a hint you never trust for equality. That script is maybe eighty lines. The reason people get it wrong isn't difficulty. It's that they run it against a live bucket.

The cutover window is where a console helps, and where rclone still beats us

Here is the concession, and we mean it: rclone is free, permanently, with no account and no vendor. It runs inside your security boundary. Your credentials never leave hardware you control, which is a stronger answer than ours and we should not pretend otherwise. Storafleet keeps credentials encrypted at rest (AES-256-GCM, per-namespace keys), and for AWS you can connect keylessly via IAM role assumption, but "the keys never left my laptop" beats "the keys are encrypted in someone's database." That is just true. If your security model says credentials must never touch a third party's infrastructure, we are the wrong tool and no amount of encryption at rest changes that. Your laptop, your keys, rclone.

rclone also composes with things a web console never will. Cron, systemd timers, Makefiles, CI pipelines, a shell script a predecessor wrote in 2019 that still runs. --dry-run and --backup-dir give you a rollback story. We don't have one as clean.

And rclone supports more backends than we do. Seventy-plus, including Google Drive, Dropbox, OneDrive, SFTP and WebDAV. We connect the S3-compatible family and any custom S3 endpoint, and nothing else. If the job is Drive-to-bucket, this is not a close call.

If your migration is one person, one weekend, and a terminal, use rclone and stop reading. If your workflow is entirely code and always will be, a CLI fits it natively and our console never will. If your budget for this is zero, the conversation is over and rclone wins. We would rather say that than sell you a subscription you do not need. Sync-only jobs, especially: if all you need is "keep these two buckets in step," rclone does that better than we do, for free, and we are not going to argue with it.

So who is the console for?

Teams, mostly. The cutover window is where a scheduled sync and a visual reconciliation earn their keep, because the failure mode we described at the top is not a tooling failure. It is a coordination failure. Someone has to look at 412 flagged objects, decide which are real, and put their name on the result. That someone is usually not the person with the terminal open, and they are not going to read your bash output.

What Storafleet actually does in that window: scheduled sync between providers so the pre-freeze catch-up runs without a human, duplicate detection so you know what you are paying to move twice, and a reconciliation view across buckets that a second person can read without being walked through it. Migrated bytes are unmetered on every tier including Free, because charging per gigabyte to move data during a cutover is a bad way to earn trust. Tiers differ on concurrent jobs and scheduled sync, not on bytes.

Cross-provider listing is also where we do real work, because the S3 API does not normalise across vendors. We handle the pagination, the checksum-field differences and the ETag weirdness, and show you the result as a table instead of a TSV you have to diff yourself.

That is the whole pitch. It is narrower than it sounds.

And here is the part we owe you on trust. We are asking you to trust us with the cutover window, the one hour where being wrong is expensive, and we cannot hand you a folder of case studies the way an established vendor can. We do not have them yet. The reconciliation logic is the product; the track record is not there to point at, and you should weigh that honestly when you decide. If a botched cutover would cost you more than you are willing to bet on a small team, wait, or run the freeze with rclone and a script you wrote yourself.

Do not use us if you need storage mounted as a filesystem. There is no FUSE surface. rclone mount, Mountain Duck or ExpanDrive do that and we do not. Do not use us if the credentials must never leave hardware you control, unless it is AWS-only and IAM role assumption covers you. Do not use us if everything lives in one cloud and always will: that provider's own console is free, current with its newest features, and authoritative in a way we are not. Do not use us if you want a decade of track record. We are a small team and Storafleet started in 2026. On "will this still exist in five years," rclone has a much better answer than we do, and anyone who tells you otherwise is selling something.

For a single person managing a few buckets with no subscription appetite, Cyberduck or S3 Browser is the honest recommendation. For consumer clouds, MultCloud or CloudFuze. We are not the answer to every migration and it would be strange if we were.

The verdict, plainly: if you have a terminal, a weekend, and no appetite for a subscription, rclone is the right answer and you should go use it. Same if your credentials cannot leave your own hardware, or the job is Drive-to-bucket, or you want a tool with a decade behind it. The console earns its keep in one narrow place: a team cutover where the freeze is scheduled, the diff has to be read by a second person, and someone has to sign their name to the result. If that's not your migration, close the tab and run rclone sync.

The engineer from the top of this post eventually got her cutover done. What she needed was not a better sync command. She needed the freeze to be a scheduled event, the diff to be something a colleague could read, and the ETag column to stop lying to her. Two of those are process. One of them is a console.