Moving the server. I planned for seconds of downtime, it took two minutes
· · Not by Prompt Alone · episode 7 · Czytaj po polsku
Nobody likes moving house. Boxes, tape, and at the very end it still turns out there was something in the attic cupboard nobody remembered 😉
Servers are exactly the same. A week ago, Claudius (my nickname for Claude) and I moved everything I had on my server to a new machine. Zakupomat went last. I'd already announced this move back with the Mac mini - "the server is next in line". So here's how it went.
A server that grew over the years
The old server was a small VPS at OVH - 1 core, 2 GB of memory. Over the years everything ended up on it - WordPress, a few small apps, static sites (including the one you're reading this on), redirects. 13 domains and services in total. Zakupomat joined them as just one of many.
And each of those things was set up "the way it made sense at the time". Debian 11 already out of support, Python 3.9, PHP from a third-party repository, most services running as the same system user. And Apache in a mode where every open connection is a separate process. That matters for Zakupomat, because every open app keeps a live connection to the server (that's how changes sync between family members' phones). We worked out that the ceiling was roughly 1,000 families active per day. Far away? Families are joining faster than I expected 🙂
The immediate trigger came from somewhere else - a PHP security patch for WordPress. I patched it and the same day I said: enough, we're moving.
The new home and what it costs
The new server is an OVH VPS-2 - 4 cores (virtual, vCPU), 8 GB of memory, 75 GB on an NVMe disk, Debian 13 set up from scratch. Cost - 38.40 PLN net a month, snapshot included. For four times more power than before. For comparison - the old server with automated backup cost 40 PLN net. So more power for less money. And that's not the end of the bill... (more on that below 😉)
I wanted Warsaw. Warsaw wasn't available for the whole line of those servers, so it's Milan. For users that's 20-30 ms of difference, i.e. nothing. Data stays in the EU, and backups stay in Paris, so in a different place than the server - just as it should be.
With everything moved in, the server is... bored. Load close to zero, 1.2 out of 7.6 GB of memory in use. And good, because that was the point - Zakupomat should have headroom for years, and the cost should stay at a level that will never force me into ads or charging users.
A plan instead of improvising
Before we touched anything, Claude wrote a plan in a GitHub issue. An inventory of the old server, stages, who does what, and rules we stick to until the end:
- the old server is the fallback - we delete nothing on it, going back means changing one DNS record,
- one service at a time - prepare, test on the new one without switching, freeze writes, switch, verify,
- after the switch the old server forwards traffic to the new one, so nobody with the old address cached on their phone hits stale data,
- passwords and keys never go through the chat or git,
- everything as scripts in the repo, nothing "by hand because it's quicker",
- Zakupomat last, and only on my explicit "deploy".
The first step, before moving anything at all, is one that's easy to forget - I lowered the TTL in DNS. It mattered most for Zakupomat, which we only moved the next day. TTL tells phones, computers and providers how long they may remember which address the server lives at. Mine was an hour (3,600 seconds), so after changing the address some people would keep knocking on the old server's door for an hour. I lowered it to 5 minutes (300 seconds) in all 7 domains. But you have to do it ahead of time - the old, one-hour value is cached too and has to expire first.
The OVH trap - just changing the zone's "default TTL" publishes nothing. Half an hour later it was still 3,600. Only saving an edit to any record pushed the change out. Worth checking instead of trusting the panel 😉
The split was simple. Everything in panels (OVH, DNS, GitHub, the password manager) and every decision - me. Scripts, comparisons and checks - Claude. And the issue was the log - Claude ticked off steps and added comments on what worked and what didn't.
I made one decision against the trend. The plan said "ideally nginx". I stayed with Apache, just in a more modern mode - 1,024 threads instead of 150 processes. Because with a single Python process, Python will be the next bottleneck, not the web server. And nginx isn't just rewriting the config of every site. WordPress keeps some of its rules (e.g. pretty post URLs) in an .htaccess file, which Apache reads and nginx simply ignores. Without any error - the site seems to work, only some URLs quietly stop working. nginx will get its turn together with a bigger change in the backend. One variable at a time.
For the same reason, on moving day we first deployed the latest Zakupomat code to the old server. So that in the evening only the server changed, not the server and the code at once.
Evening one: five services in ten minutes... and a wildcard
The first evening went faster than I expected. Base system, backups, static sites. Then WordPress and four small apps - five services in about 10 minutes, each with a break of a few seconds to a minute.
Except that earlier, with the static sites, we'd switched the DNS record for "everything else" (a so-called wildcard). And BOOM! Five services that, according to Claude's list, had their own records went along with it to the new server before the new one even knew they existed. From the outside you can't tell the difference - a name lookup returns exactly the same thing. About 10 minutes of downtime, fixed without touching DNS - for a while the new server forwarded those names to the old one.
The lesson that went into the brain: inventory what the server actually serves, not what's written in the description. The list of names in the server config is the truth. The DNS zone is only a description.
Evening two: Zakupomat and two minutes
The second evening, after my "go ahead with the move", Zakupomat went over in one script. Stop the API on the old server, final database backup, import on the new one (217 families), start, old server as a go-between, backups running on the new one straight away. Planned downtime - seconds.
Actual - about 2 minutes.
On the old server the API config file was a plain copy, not a link to the file the script was editing. The script changed a file the server didn't read at all. Same mistake as with the wildcard, just somewhere else - an assumption about how things "should be" instead of checking how they are.
And guess who happened to be adding products to the list in those two minutes? Me 😂 11 additions got "Server error (HTTP 503)" and were lost - the app's offline queue covered checking items off, but not adding them. Plus a message that says nothing and doesn't suggest what to do. It ended with two issues - adding should go through the queue too, and the message should speak human and offer a "retry" button. Moving house as a free resilience test for the app 🙂
Five minutes later the first phones were already connecting straight to the new server. Over the whole next day only 10 requests went through the old one - that's the short TTL at work. The days after - only scanners and bots.
Backups that saw too much
During the move I decided that everything gets off-server backups, not just Zakupomat. A new container in Object Storage, a new user with access "only to its own container".
And that's when it turned out that "only to its own" didn't work. Neither for the new user nor for the old one. In OVH an empty access policy doesn't mean "nothing". Access comes from the role itself, and the role covers all containers in the project. Zakupomat's key had been able to see everything for three weeks.
Was it a problem? Honestly - until then, no. For three weeks there was nothing in that place except Zakupomat's backups, so "everything" and "only Zakupomat" meant the same thing. The problem started exactly when I added a second container with backups of the other services. From that moment, broad access mattered. And the decision in the brain had said from day one that the user was "only for it". Nicely written down, never checked.
Now every key has a policy for its own container only and can't delete previous versions of backups. Even someone with the key from the server can't wipe the history. You can't write isolation like that down once and forget it - you have to test it from the side it's supposed to defend.
The second thing is backup monitoring. Until now it was watched by a monitor in GitHub Actions that asks the server about its state every 15 minutes. Except when the server is down, there's nobody to ask... So the other services' backups are now watched by Healthchecks - after every successful backup the server checks in, and if there's no check-in within a day, I get an email. It catches a dead server, a disabled timer, and a backup that quietly stopped running.
And in the attic cupboard...
While tidying up I also looked at the second server. Yes, I had a second VPS. And it turned out that for about a year there had been nothing on it that anyone used. No domain pointed at it, the last certificate renewal - September 2025. I moved the old apps off it last autumn... and the server stayed. And every month the bill politely arrived - 21 PLN net, snapshot included. Over a year that's more than 250 PLN for nothing 😮
After the move the server bill drops from 61 PLN net a month for two to 38 PLN for one - four times more powerful.
Everything from it is archived on the Mac mini, services stopped, and after a safety month I'm cancelling it. The old main server is headed the same way.
There was a second find too. The code of one of the small apps didn't exist on GitHub - the repo had only an "Initial commit", and the working version lived only on the server. Remember the Mac mini? If it's not in the repo, it doesn't exist. Now it is.
Where the agent got it wrong, and where I did
Honestly - Claude made mistakes. The DNS record list without the wildcard. Two database users it declared "dead", when those apps' deployments used them to change the database. It assumed an app didn't read its config file itself, and it did. The API config file that turned out to be a copy.
But every one of those mistakes was caught by a check right after the step, not by luck. The thing that paid off most was one small tool - a script that asks the old and new server for the same addresses and compares the answers before anyone touches DNS.
And me? I paid for an empty server for a year and for three weeks believed in isolation I'd never checked 😉
What stays
- a server on Debian 13, every service on its own user and with limited permissions,
- off-server backups for everything, with a weekly restore test from the container, not from the disk,
- a repo with the configuration of the whole server - the next move will start from a script, not from memory,
- two lessons in the brain: "inventory what the server actually loads" and "isolation written in a decision isn't isolation until you check it",
- two issues that will make Zakupomat more resilient to the next outage.
The move took two evenings. Without the agent it would probably have taken me two weeks... and I probably wouldn't have done half of this tidying up.
If there's a server in your company that "everyone knows what's on" - it's worth checking before you start moving it. Taking inventory from what the system actually does, not from the documentation, is half the success of any migration. That's exactly where we start at ANSLAN - we rebuild the knowledge about the system before we touch anything.
Zakupomat is free and ad-free. On its new server for a week now 🙂