Like some of you have already noticed, the instance has been down for the last week and a half. I documented the problem and the process of bringing it back online in a thread on our Discord, so if you are curious about that I’ll redirect you there.
The TL;DR is that the instance ran out of disk space, so the database crashed. No database, no Lemmy.
I solved it by moving our 30GB of images to a separate, much cheaper storage (we moved from 3,00€ / month down to just 0,02€ / month for the image storage!), freeing up a bunch of space for the database. This should keep us going for a while and allow us to scale much better in the future.
The new host
The new image host we are using, is an object storage located on a separate machnie from the Lemmy server (while previously images resided on the same server as the database and the instance). Because of this, you are likely to experience some milliseconds of delay with the loading of new images, because some back and forth between Lemmy and the image server needs to happen before you can see it (Lemmy downloads the image, sends it to the image storage, image storage returns a link to the image to Lemmy, THEN you can see the image. It takes a while).
Next moves
While I’m on a roll with the Lemmy updates, later today I plan on updating our instance to version 0.18.5 of Lemmy. This should give us even more stability and better uptime in the future, but might temporarily break user flairs in !pcm@lemmy.basedcount.com.
I am terribly sorry for the prolonged downtimes and I really appreciated all the people who joined our Discord server asking if they could somehow help or simply showing care for our work on the instance.
Please remember that this is mostly a solo project of mine, where I am left handling both the server admin side as well as the community facing one. It’s a lot for one guy to deal with, I’m sure you’ll understand.
EDIT: I’ve succefully updated the instance, everything seems to be working fine. Let me know if something feels odd or buggy.
You must log in or # to comment.
Glad to see it back up, I was wondering if something was wrong with my computer
Happy Thanksgiving (U.S.) belatedly to posters
Way to finally make the thing do the thing, finally. Or whatever.
deleted by creator
post the reason here
I literally wrote it up there:
The TL;DR is that the instance ran out of disk space, so the database crashed. No database, no Lemmy.
If you care about some more details you can read what I wrote on Discord. If you don’t carry on. But whatever, for the sake of completeness here’s the wall of text.
Context: the following was written on 26/11/2023 18:23:52 UTC, before I started working on the problem. The migration to the S3 took about 4 days.
What the hell is going on?
The instance is currently offline because we ran out of space. You lot shitposted too much and filled the database. Expanding the storage isn’t trivial and is significantly expensive, so we are using this outage to do some much needed maintenance and make the instance more affordable for us to run in the future, as well as more resilient to similar kinds of issues.
Yeah very cool but what does it mean? In English?
The problem
Right now the instance is lives on two drives: a 15GB one containing the database and a 30GB one containing images uploaded by other users or synced from other instances. The former filled up, so the Lemmy backend is crashing on startup. The server is now stuck in a loop of starting, realizing it can’t write to the database, crashing and restarting. The only way to exit this loop is to either spend money and buy extra space or to delete some old data.
Our solution
The more intuitive solution would be expanding the database drive, however that’s just too expensive and not really sustainable in the long run. Instead, we are going to transfer the image host to a significantly cheaper host, free up the 30GB drive and move the database there. This solution is, however, a bit more complicated and takes some time.
If you speak computer shit, what I just said means we are going to move the image host (pictrs) to a remote S3 object store, right now it’s located in the same filesystem as the Lemmy server.
