Alt account of poVoq@slrpnk.net here.

Our instance is currently down and I can’t get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can’t work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can’t access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy’s law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this “opportunity” to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

  • pjusk@lemmy.dbzer0.comEnglish
    10·
    1 day ago

    Thanks for all that you do for us and this community! A little downtime won’t hurt anyone. I would also be in gavour of PieFed migration 💙

    • solo@fedia.io
      4·
      8 hours ago

      I totally agree with everything you said :)

      I just created an account in PieFed and it looks very interesting because there is the option to create your own feeds! I had the wrong impression that you can only choose from the existing ones. This is so exciting!!!

  • ocean@lemmy.selfhostcat.comEnglish
    19·
    2 days ago

    I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.

    Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?

    • Kris@feddit.orgOPEnglish
      121·
      2 days ago

      We have a small write up about the hardware on our wiki, but it is also down right now.

      I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.

      One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.

  • Initiateofthevoid@lemmy.dbzer0.comEnglish
    35·
    3 days ago

    Try not to stress too much about it while you’re away! If nothing can be done, nothing can be done. Solarpunks need to be resilient to things like this anyway. Community over convenience.

    • Kris@feddit.orgOPEnglish
      20·
      3 days ago

      Thanks, I am trying, but it really is quite annoying to lose access like that.

    • evasive_chimpanzee@lemmy.worldEnglish
      40·
      3 days ago

      Seriously, I think a big part of solarpunk ethos is combating the notion that everything has to always be available 24/7. Society pays a lot to deliver every convenience like fruit out of season from the other side of the world.

      • stabby_cicada@lemmy.blahaj.zoneEnglish
        18·
        3 days ago

        And on the punk side of things, anybody familiar with volunteer community action, cooperative events, whatever, knows that sometimes a person doing shit for the community fails to do the shit they were supposed to do because some other shit happened in their life. It happens. We just move on to the backup shit-doer-person or find a workaround for the lack of shit.

        And this is the cool thing about a distributed network like Lemmy. If a giant centralized database like FB goes down, its users can’t do anything. If our favorite Lemmy node goes down, we can log into other nodes until our home base is back up.

  • Sergio@lemmy.worldEnglish
    34·
    3 days ago

    Sorry to hear that! My main account is on slrpnk.net. But don’t let it stress you out, “solarpunk” says “punk” which implies “DIY” which means this sort of thing happens. Much better to have an instance run by an admin who truly cares, rather than some soulless corporation.

  • gandalf_der_12te@discuss.tchncs.deEnglish
    21·
    3 days ago

    This is actually great timing, i’d say. Just now most of the northern hemisphere is starting into summer. At least i deeply feel it on a personal level that summer is arriving. I take this as a gift from the universe that directs me towards touching grass, and going swimming. Enjoy your summer, you all! Connect with your local community, if you can.

    • Kris@feddit.orgOPEnglish
      15·
      3 days ago

      Glad that at least someone sees a positive side of it, because I sure don’t… well at least it gives me a bunch of ideas how to avoid such situations in the future 🤷

    • Draconic NEO@lemmy.worldEnglish
      1·
      2 days ago

      Yeah me too, I’d wanted to for a while but never got around to it. Now I probably never will (unless they get the servers back up at some point in the future).

    • Five@lemmy.cafeEnglish
      9·
      3 days ago

      You’re very welcome! Kris@feddit.org / @poVoq@slrpnk.net is an incredible admin, and up until now has declined donations, funding the entire instance several years now from his own pocket.

      This current downtime is the exception that proves the rule. I hope he’s not stressing out too much. We’ll all be ready to jump right back into it when he gets back.

      • Kris@feddit.orgOPEnglish
        10·
        2 days ago

        Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.

        • Draconic NEO@lemmy.worldEnglish
          2·
          2 days ago

          Wouldn’t that mean the instance would completely restart? I mean I don’t know much about piefed but logically it seems like the DB structure would be incompatible. Thus “Migrating” to piefed means a complete reboot, all posts, all users, all content just bye bye.

          • Kris@feddit.orgOPEnglish
            8·
            2 days ago

            Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.

            Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.

            If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.

            • Draconic NEO@lemmy.worldEnglish
              4·
              2 days ago

              Oh awesome, I mean I’ve heard of other platforms like sublinks offering database migration but I didn’t think it would be feasible with piefed since unlike sublinks it’s very different than lemmy.

          • Kris@feddit.orgOPEnglish
            8·
            2 days ago

            Lemmy has a lot of individual parts that don’t interact very well with each other, especially the image host part. Futhermore the main UI is quite a mess and we were thinking of switching to an alternative already, but this would further increase the “too many moving parts” issue. Piefed on the otherhand has an integrated and very lightweight UI, which also has some nice additional filtering and moderation features Lemmy currently lacks.

            And I personally feel more at home with the Python codebase, as it allows better troubleshooting and more standartized (Flask) tooling. The Rust codebase of Lemmy has a lot of obscure custom stuff and the error messages are extremely obstruse from a sysadmin perspective.

            And looking at the performance metrics of Lemmy, the main limiting factor seems to be the Postgres database anyways, so the theoretically slower Python codebase of Piefed should not have much impact.

        • PirateFrog@lemmy.dbzer0.comEnglish
          9·
          3 days ago

          That’d actually be pretty rad if it’s doable, especially now that the API for piefed was released and mobile apps are already beginning to support it.

    • glaber@lemm.eeEnglish
      3·
      3 days ago

      Yeah! It’s definitely worth adding to my list of migration contenders now that lemm.ee is closing and I have to jump ship

  • PirateFrog@lemmy.dbzer0.comEnglish
    301·
    3 days ago

    Alt of ProdigalFrog@slrpnk.net chiming in, thanks for giving us the sitrep, poVoq!

    This is an unusually unlucky set of circumstances, but for the Solarpunks who find their way here, let’s try to make the best of it. If you don’t already have an alternative account somewhere, you may want to create one to use until we’re back up and running. lemmy.dbzer0.com, sopuli.xyz, or lemmy.cafe are some solid choices to consider. After that, you may want to favorite poVoq’s post here so you can check back on any news.

    Even without our home base, we can still share the solarpunky stuff we find and create around the threadiverse (though as Tofu mentioned, maybe this could be a time to take a breather to be refreshed when we come back 😄).

    Besides, if the past is any indication, time will fly by and we’ll be back before you know it!

    • Kris@feddit.orgOPEnglish
      10·
      3 days ago

      Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.

      • mad_lentil@lemmy.caEnglish
        2·
        2 days ago

        Might it also be possible/feasible to send an “email blast” to everyone with a slrpnk.net account? It would probably be quite complicated to send that many emails at once without getting blocked by the major email providers, but hypothetically we’ve all registered with emails…

        Just spitballing ideas for the future. Everything’s fine now. Gives me an excuse to spend a little more time on the Canadian end of the fediverse. :)

        • Kris@feddit.orgOPEnglish
          4·
          1 day ago

          As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered would be another problem of mass-mailing.

          But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.

      • tenchiken@lemmy.dbzer0.comEnglish
        5·
        3 days ago

        Not sure about your area, but a wireguard accessible OOB connection is a great piece of kit to keep handy. I use a cheap 768kbps SIM in an Ethernet connected switch into my personal systems. It’s saved my skin numerous times.

        I’m sure this is obvious, especially in hindsight, but just mentioning because the existence of IoT LTE data plans for a minimal fee ( $100/year for me in Midwest US) was NOT obvious to me until 2 years ago.

        • Kris@feddit.orgOPEnglish
          2·
          3 days ago

          Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn’t work over such a small pipe either.

          • MelodiousFunk@startrek.websiteEnglish
            6·
            3 days ago

            This is something that even larger corps struggle with. My old company would buy some other company, lay off a bunch of people, others would quit, and then it was shocked Pikachu faces all around when That One Thing stopped working in an office they turned into a ghost town and, well, no, nobody was going to be there until Monday morning to power cycle etc. True lights out/OOB reachability is WORK. And there’s always going to be a SPOF somewhere that requires hands on-site.

            Do what you can, when you can. And thank you for all that y’all have done so far.

          • curbstickle@lemmy.dbzer0.comEnglish
            2·
            3 days ago

            For that, I’d highly recommend a power conditioner with outlet controls (and preferably sequencing).

            Most are pretty expensive, but you can get something like a surgex squid or a gude expert power control for a few hundred here and there, which will give you great data on power usage as well. You can also grab an older model with serial and/or relay controls, and use a pi or whatever is handy to trigger it. Actually have a setup like that in a few racks for clients (mostly orange pis and a couple of Asus tinker boards).

            Hope your trip goes well!

  • khaliso@lemmy.worldEnglish
    7·
    3 days ago

    Thanks for the heads up! My Main account is in slrpnk, so I’ll set a reminder in an month to switch back. :)