INaudio: $500K a year out of a two petabyte bill

The challenge

When INaudio returned to independent ownership and moved off Spotify’s infrastructure, it inherited its own cloud bill for the first time in four years. More than two petabytes of audio, artwork, and metadata, spread across 120+ microservices, all of it built to somebody else’s cost assumptions.

The first month after the cutover, we changed nothing. Deliberately.

Internally that phase was called seeing how the shoe fits, and the mantra was observe everything, change nothing, ensure everything runs smoothly. It is tempting to skip. A team that has just finished a migration is at peak familiarity and peak confidence, and there is a real cost to sitting on your hands for a month while a bill you can see runs. We sat on our hands anyway, because optimising a system you have not yet watched under load is how you turn a cost problem into an availability problem.

In September, we started making changes.

What we did

Rewriting the infrastructure in code that was actually true

The first move was to replace the entire infrastructure definition, from CloudFormation to Terraform.

Before

CloudFormation

AWS proprietary, so tied to one cloud. Drifted years away from the environments they described, which the migration had already proved the hard way.

After

Terraform

Cloud agnostic, cleaner, and replicable. The definition and the environment now agree, and that agreement is what every later saving was found through.

The reasons usually given for this are portability and avoiding vendor lock-in, and both applied. But the reason it mattered here was more immediate: we had just spent eight months discovering that the old templates no longer described reality. Rewriting them meant the map matched the territory again. You cannot find waste in an estate you cannot accurately describe.

We also stood up a second non-production environment built to current standards, which gave the legacy code that had been gathering dust somewhere safe to be modernised.

Doing that work is what surfaced everything below. None of it was found by looking for savings. It was found by writing down what was actually there.

The report that cost 19 times what it calculated

• Before $15,000
• After $15

A 99.9% reduction. The lower bar is drawn at its minimum visible width. Truly to scale it would be about half a pixel.

One monthly report calculated royalties owed for a niche category of listening event. The infrastructure supporting it cost about $15,000 every time it ran.

The first time the new team ran it, the report came back saying clients were owed $800.

The infrastructure cost was nearly nineteen times the total value of the number it existed to produce. Nobody had ever put those two figures next to each other, because the report’s cost and the report’s output lived in different systems and were nobody’s shared responsibility.

• Cost to produce $15,000
• Value calculated $800

The cause was in how the data was arranged. Every run queried a live table holding historical records going back to 2018, so the cost of the report grew with the entire history of the business rather than with the month being reported.

We discontinued that reporting method and decommissioned the infrastructure behind it. The report now costs $15 to run.

Two petabytes, three moves

Storage was the larger prize, and it came apart into three separate problems.

1. Step 1

   Deduplicate

   We took the S3 inventory report, loaded it into an Athena table, and collapsed files sharing an ETag. Matching on the checksum rather than the filename means renaming and repackaging cannot hide a duplicate.

2. Step 2

   Clear the hidden versions

   Nearly a petabyte of data was invisible in the interface but fully billable. Object versioning had been switched on at some point in the past, and it silently defeated the 30 day lifecycle rule.

3. Step 3

   Let tiering sort it

   S3 Intelligent Tiering now moves files between Frequent, Infrequent, and Deep Archive Instant Access automatically, based on how they are actually used. No lifecycle policies to write, and none to maintain.

Combined effect on the storage bill: more than $500,000 a year.

The second of those is worth dwelling on, because it is the most instructive failure in this whole engagement.

Someone, at some point, had enabled object versioning. The lifecycle rule was supposed to move a file to cheaper storage or delete it after 30 days. With versioning enabled, it did neither. It expired the current version and hid it. The file stayed exactly where it was, on exactly the expensive storage class, and kept being billed, while the interface showed a clean and apparently well-managed bucket.

The cost signal and the visual signal pointed in opposite directions, and the visual one was the one people saw. Close to a petabyte accumulated behind that discrepancy. It is a good argument for auditing an estate against its inventory rather than against its console.

The third step involved a choice worth explaining. The obvious tool is the S3 Storage Class Analyzer, which is free to use and tells you what to move. Acting on its advice is not free: it is pay-to-move, charged per object, and across 400 million files that transfer charge is the entire saving. Intelligent Tiering charges a small monitoring fee instead and then does the sorting itself, continuously.

For audiobooks that is close to an ideal fit. A popular title is streamed constantly and a back-catalogue title is barely touched, and which is which changes with a review, a screen adaptation, or a new release by the same author. Tiering follows real access patterns as they shift. A lifecycle policy would need someone to predict them in advance and rewrite the rules whenever they were wrong.

The results

• $500K+taken out of the annual cloud bill
• ~$1Mprojected once every change lands
• 99.9%cut in the cost of one monthly report
• ~1 PBof hidden versioned data recovered

More than half a million dollars a year, from changes made in the months after taking ownership. With everything currently in flight implemented, we expect that to reach roughly a million annually.

None of it came from negotiating a better rate. It came from the infrastructure being written down accurately, one report’s cost being compared to its own output, and an inventory being audited against the bill rather than against a screen.

What we would do next is leave S3 entirely. Outbound data transfer is the dominant cost in audiobook distribution, and past a certain scale that is a structural property of the platform rather than a configuration to tune. The savings above are real and they are also, eventually, a ceiling.

The rest of this engagement

This was one of three workstreams. The infrastructure existed to be optimised because Tarmac had moved the entire platform off Spotify with one hour of downtime, and the same period produced an agentic support workflow that cut ticket triage from two days to twenty minutes.

Tarmac is INaudio’s technical team, not a vendor engaged to run an audit.