As you might have heard several Lemmy instances have been attacked via a security vulnerability in the browser frontend related to custom emoji.

While SLRPNK was vulnerable to it, we seem to have not been actively targeted and I took the instance down as a precaution as soon as I learned about it.

I have applied all the currently known mitigations, which means that everyone got logged out of their account and needs to log back in manually.

As of writing this the API is working again and can be used with apps like Jerboa safely.

I am still contemplating if I want to re-enable the web frontend now or wait for a release that fixes the issues found.

Edit: the main issue was fixed and I restarted the web ui with it.

  • RiverGhost@slrpnk.net
    9·
    1 year ago

    Ah, so that’s why it was down. Thank you for the quick reaction.

    Some of the apps * seem to have had trouble when you’re logged out remotely, mine still seemed logged in, but kept giving me a a vague ‘network error’. I had to manually log out and then log in.

    * Tried Jerboa, Liftoff and Thunder with similar results.

  • Steve@slrpnk.net
    2·
    1 year ago

    I’ve noticed the instance isn’t working smoothly for me today, unfortunately. Nearly every time I reload the feed, specifically on mobile web u but also on desktop, I’m logged out and have to log back in. Additionally I’m not sure if this related to the API but 2 apps I’m using on testflight have stopped showing my “subscribed” feed and only show “local” or “all”. Not sure if this instance specific but these issues started this morning (7/10) for me so thought it’s worth noting.

    • poVoq@slrpnk.netOPM
      1·
      1 year ago

      Did you try logging off manually from the apps and logging in again?

      Edit: sorry I didn’t want it to sound like a snarky tech support comment. But the apps seem to have issues with being logged out forcibly by the server.

      As for the web-ui. No idea. nothing changed substantially.

      • Steve@slrpnk.net
        1·
        1 year ago

        Not yet, I would need to stop testing them and then reset the testflight, that however doesn’t seem like it would explain the constant logging off from the web ui as I’ve had to log back in to respond to this

        • poVoq@slrpnk.netOPM
          2·
          1 year ago

          The only thing I can think of is that you still have problems with cached JS code from pre-0.18.0 times. Try force reloading the page via CTRL+F5 and see if that helps.

          For me the web-ui is not showing any issues in Firefox, so I have no idea how to reproduce or try to fix it.

          • Steve@slrpnk.net
            1·
            1 year ago

            I’m currently not at home using the mobile web ui on Brave which ordinarily doesn’t give me these issues. I’ll see about deleting the cache in the apps and troubleshoot more when I get a chance. I’m sure it’s not an issue caused by your work but does seem like the latest update could be a little buggy so I’ll try resetting things asap

  • thisfro@slrpnk.net
    2·
    1 year ago

    Thank you for the quick action! Do you have a channel on another service, where one could see the status of something like this? Maybe matrix, mastodon or similar? If not, that is fine too of course! But maybe there is already anyway

    • poVoq@slrpnk.netOPM
      2·
      1 year ago

      There is a Matrix channel for instance admins if you mean that?

      For status of the infra, we have: https://health.f-hub.org/ (slrpnk.net is at the bottom), but that could probably be improved now that the Lemmy backend has support for Prometheus monitoring.

      • thisfro@slrpnk.net
        1·
        1 year ago

        I was thinking more of a channel where you (and other admins) could make announcements such as this post, which is readable when the instance has issues/is. So we know what is going on (not only that it is down, but also why) and could also maybe help out if needed :)

        • poVoq@slrpnk.netOPM
          1·
          1 year ago

          Hmm, yes that might be good. For now there is also my personal Fediverse account: https://outmo.de/kris but it runs on the same infra so if there is a more serious issue it will be down too.

          But generally speaking if there are issues that effect more than slrpnk.net other Lemmy instances will also be down or at least talk about it.

          I think the next step is to complete my plans for the infra that will allow easier access for external sysadmins so that we can reduce the bus factor. I’ll keep you all posted when this will be possible.