Subject to change
SAKbible's entire memory lived on a free tier. One afternoon the provider exercised the only clause it ever had. What died, what survived, and why the next loss costs an afternoon instead of a weekend.
the day a free tier exercised its clause
Every free tier runs on the same one-line contract: subject to change. One afternoon somebody at the provider changed it. The workspace holding SAKbible’s entire memory — every vector, every secret, every cached model — was disabled. Not by me, not for anything I’d done, and with no explanation owed.
SAKbible is the tool that refuses to answer a question without a citation. That afternoon it couldn’t have produced a citation with a gun to its head: the library it quotes from was gone, along with the building it lived in — and the building was never mine.
What was in the building
- The parsed standard — the corpus itself, every section chunked and ready.
- The embedding vectors for all of it.
- The cross-reference graph that lets a citation point at the right section instead of a neighboring one.
- Secrets: the API keys, the ones that are never in git and therefore live exactly where outages can find them.
- A model cache holding two models whose only remarkable property is how long they take to download.
The total weight is embarrassing. A handful of megabytes. The corpus itself was never in danger — small files, all mirrored locally, all of it. What the outage actually took was the live copy, the one the app reads, plus the setup knowledge that existed nowhere except my head. None of it was large. All of it was load-bearing.
The rebuild
Recovery was never in doubt, which is the embarrassing part. The sequence:
- New workspace.
- Re-upload the volume.
- Re-set the secrets — which key belongs to which app, the part that is written down nowhere because it was obvious until it wasn’t.
- Re-warm the model cache. Both downloads. The slow ones.
- Repoint the apps.
A day of archaeology in my own configs. Rebuilding your own infrastructure is the only audit that can’t be gamed: it doesn’t read your documentation, it interviews you, and it grades on whatever comes out of your head at the hour you happen to reach step 3.
What changed
Nothing dramatic — that’s the point. Everything I care about now exists in three places: git, a local mirror, and a volume I can rebuild from a manifest with checksums attached.
The manifest is the part people skip. A backup you haven’t restored is an opinion; an opinion with a checksum is at least a testable opinion. The corpus fits in one rsync command, and the command is written down where the next me can find it without going archaeologist first.
The scripts already knew
When something deployed is down, I run three diagnostic scripts: check the environment, check the network, check if it’s warm. Somewhere in the middle of the rebuild I noticed the sequence is the stages of grief with shebangs. There is no fourth script. The fourth stage is rewriting the ingest script at an hour I won’t admit to.
My README’s stop instructions end with a comment about the bracket trick in pkill — the one that keeps pkill from matching its own command line and taking your shell down with the server. The comment is written in the tone of a man who has lost that fight before. That comment is load-bearing too.
After
The next time a workspace vanishes — and one will; the footer said so in writing — it costs an afternoon and a checklist instead of a weekend and a grudge. I’d like to tell you the experience made me calm. It made me prepared, which is what calm looks like on a budget.