It’s almost caught up. My apologies. I am trying to get it fixed permanently.
It’s almost caught up. My apologies. I am trying to get it fixed permanently.
:( I’m working on it
ok - the queues are processing again. I will work on a more permanent fix after dinner
ok - rabbitmq started having prroblems with the delivery queue again. I got it going again. Those messages should be delivered soon.
I am not 100% sure. Fedia.io is running on a beast of a server, and so long as it’s working correctly, it should be able to deliver it instantly. But that doesn’t mean that the receiving servers are able to consume and render them that fast.
Once it is more reliable, then I’ll agree, but thank you
there isn’t an inteval per se - I don’t yet know why it’s not immediate. It’s possible that the delay is on the receiving side - the server that fedia.io is quite substantial and unless there is some sort of bug, the processing should happen immediately.
Outbound federation was indeed broken. I fixed it, but there was a huge backlog the server had to work through, which took about 12 hours to complete. I just checked and everything appears to be working ok. I am going to create some automation that will detect and alert on this (or other issues) happening in the future.
Not entirely. It looks like the rabbit issue was only impacting one of the queues (“deliver”), though I would have expected that to impact things like microblog too. All I can say with clarity is that the instance was operating in a very unhealthy state.
The queue appears like it’ll take several hours to flush, but it’s working.
I found the problem and the queues are running now. I will make a thread about it.
Thank you for the problem report. I will be looking into this later today.
Do you know whether there are any users on kbin.melroy.org that subscribe to your magazine? If not, that will explain why your post did not show up until you searched for it.
Which magazines/communities are you seeing problems with?
The only posts I see there are from the moderator - do they even permit others to post to that community?
That would be my first guess. I’ll try some tests with LW To make sure it’s working correctly
I did end up disabling registration due to spam. I can either open them up at a time you’re free to try again or I can manually create an account for you.
Note that even once I get this fixed, there will inevitably be another problem crop up. Posting here is fine, but an email to jerry@infosec.exchange or a ping to @jerry@infosec.exchange (my mastodon account) would probably be faster (note, when federation breaks here, messages to @jerry@infosec.exchange wouldn’t get through from fedia.io…