WordPress CDN failure modes

Your offload plugin is lying to you about deletes

There is a category of WordPress plugin whose entire job is to move your media library to a CDN and then delete the local copies so you get your disk back. That second half is the part people buy it for. It is also the part that is genuinely dangerous, and most of the plugins in this category do it on evidence that would not survive five minutes of scrutiny.

Here is the shape of the bug. The plugin uploads a file to the CDN. The upload call returns without throwing. The plugin marks the file as offloaded and deletes it locally. Disk reclaimed, everyone happy.

The problem is the middle step. "Returned without throwing" is not the same as "the bytes are on the CDN and retrievable." Those two statements come apart more often than you would like.

The ways an upload lies

Uploading a file to a remote service is a network operation, and network operations have a much richer failure vocabulary than success and exception:

  • The truncated write. The connection drops partway. Some storage APIs will happily accept and store a partial object, and return you a perfectly cheerful 200 for it. The file exists. It is 40% of an image.
  • The rate limit that looks like a success. You are pushing 800 files a minute during a bulk migration. The API starts shedding load. Depending on the provider, what comes back can be a 200 with an error body, which is a response no naive client checks.
  • The timeout on the client side. PHP gives up waiting. The upload actually completes on the server two seconds later. Your code has already recorded a failure, or worse, already moved on. Now local state and remote state disagree and nothing will ever reconcile them.
  • The write to the wrong place. A path with a character the provider normalises differently than you do. The upload succeeds. The file is at a URL your rewriting logic will never generate.
  • The permission that changed underneath you. Someone rotated a storage key or narrowed a zone's write scope. Every subsequent upload fails identically and silently, and the plugin cheerfully deletes local copies for the entire duration.

Any single one of these is survivable if you notice. The reason this class of bug is expensive is that it is silent and it is bulk. Nobody offloads one file. They click "Offload All" on a library that has been accumulating since 2014, walk away, and come back to a green checkmark.

What verification actually means

The fix is not clever. It is just unglamorous enough that plugins skip it: before deleting a local file, confirm that specific file, by name, is retrievable from the CDN.

Not "the upload call returned." Not "the batch reported success." Not "we have a database row saying this was offloaded." Ask the CDN for the object and check what comes back.

This costs you a round trip per file, which is why it gets skipped. On a bulk operation across a large library that is real time. It is also the only thing standing between a customer and a permanently missing image, and the trade is not close. Time is recoverable. The file is not.

Two design consequences fall out of taking this seriously:

Deletion has to be opt-in. If the destructive path is on by default, then every misconfiguration, every expired key, every zone typo becomes data loss instead of an error message. Off by default means the worst case for a fumbled setup is that nothing happens, which is the correct worst case.

Metadata gets written last. There is a tempting ordering where you clear the local record first and then do the remote work, because it makes the code simpler. It also means that when the remote work fails you have already destroyed the only map back to the original state. Read metadata first, do the remote operation, check its actual return value, log orphans, and only then let anything be cleaned up.

The other half nobody rewrites

While we are here: the same category of plugin routinely rewrites the main image URL and forgets the srcset.

WordPress has generated responsive image sets for years. A single image in a post is really a main src plus a list of alternate sizes the browser picks from based on the viewport. If you rewrite the src to point at your CDN, delete the local files, and never touch the srcset, then the page looks perfect on your desktop and every one of those alternate URLs 404s on a phone.

The failure is invisible to the person who did the migration, because they check their work on the machine they did it on. It surfaces weeks later as "some images don't load on mobile sometimes," which is close to the least debuggable sentence in this business.

What I actually do about it

I wrote Walter Offloads after watching the popular options fail in production: a 787,000-file news site hitting the memory ceiling because its offload plugin loaded the entire index on every page view, and responsive images going blank across a whole site because nobody had rewritten the srcset. Both of those are regression tests now.

It verifies every file before it deletes anything, rewrites the srcset explicitly rather than hoping, and keeps its query cost proportional to the page being rendered instead of the size of your library. It also publishes the things it does not cover, because a tool whose pitch is "refuses to lie to you" does not get to be selective about that.

If you are running an offload plugin right now and you have ever clicked "delete local copies," here is a worthwhile ten minutes: pick twenty image URLs at random from old posts, load them, and load their srcset variants on a narrow viewport. If they all resolve, you are fine and you have lost ten minutes. If they do not, you have found it now instead of the day the client does.

I write these when something in the work turns out to be worth writing down. If you're hitting the thing described above and want a second set of eyes on it, tell me what's not working.

← All notes