Part of this might be my general disdain towards sysadmins who don’t know the first thing about technology and security, but I can’t help but notice that article is weirdly biased:
Over the past couple of days, these unsung heroes who keep the internet up and running flocked to Reddit to bemoan their soon-to-be increasing workload.
Kind of weird to praise random Reddit users who might or might not actually sysadmins that much for not keeping up with the news, or put any kind of importance onto Reddit comments in the first place.
Personally, I’m much more partial to the opinions of actual security researchers and hope this passes. All publicly used services should use automated renewals with short lifespans. If this isn’t possible for internal devices some weird reason, that’s what private CAs are for.
I’m on the side of “automate it all and stop whining”, but I do think it’s important not to so readily dismiss the thoughts and opinions of those this directly affects in favour of the opinions of the security researchers pushing the change.
There are some legitimate issues with certain systems that aren’t easily automated today. The issue is with those systems needing to be modernised, but there isn’t a big push for that.
I’d be more concerned as well if this would be an over-night change, but I’d say that the rollout is slow and gradual enough that giving it more time would just lead to more procrastination instead, rather than finding solutions. Particularly for those following the news, which all sysadmins should, the reduction in certificate lifespan over time has been going on for a while now with a clear goal of automation becoming the only viable path forward.
I’ll also go out on a limb and make a guess that a not insignificant amount of people only think that their “special” case can’t be automated. I wouldn’t even be surprised if many of those could be solved by a bog-standard reverse-proxy setup.
Exactly. My “special case” took a little more care, but it works completely fine. Here’s my setup:
TCP proxy at edge -> wireguard tunnel using SNI to route to the right service
reverse proxy that handles all TLS for all services on its device (renewals and crypto)
HTTP services behind a firewall that only communicate w/ proxy
I have my router configured to resolve DNS to #2, so I don’t need to hit the WAN to access local services over TLS, and it uses the exact same cert as WAN traffic and the browser is happy.
This is about as exotic as I can think of, and it still works just fine for TLS renewals, and it’s 100% automated. I do need to leave HTTP open (it only serves acme endpoints, so whatever), but I could also close that down and have the renewal process open that temporarily if needed.
The only special case I can think of is a device that rarely turns on, which is incredibly rare these days (you’d generally have an always-on gateway that uses self-signed certs or something for those devices that stay off).
Usually the systems that need to be modernized are working, so nobody wants to invest in a new system that may require retraining the people that may be impacted.
Then there’s some systems with integrations that may also require replacing so the integrations can continue to work.
Even then, there’s always a good possibility that the automation fails, especially in the first few iterations of trying to sort out the kinks, and third party automation tools aren’t perfect either. That’s another tool to have to update and maintain once all is said and done.
I’m not trying to rail too hard against the changes, but the impact is especially felt by the people managing the systems, who’s most likely getting more work tacked on to their workload of putting out fires behind the scenes.
Part of this might be my general disdain towards sysadmins who don’t know the first thing about technology and security, but I can’t help but notice that article is weirdly biased:
Kind of weird to praise random Reddit users who might or might not actually sysadmins that much for not keeping up with the news, or put any kind of importance onto Reddit comments in the first place.
Personally, I’m much more partial to the opinions of actual security researchers and hope this passes. All publicly used services should use automated renewals with short lifespans. If this isn’t possible for internal devices some weird reason, that’s what private CAs are for.
I’m on the side of “automate it all and stop whining”, but I do think it’s important not to so readily dismiss the thoughts and opinions of those this directly affects in favour of the opinions of the security researchers pushing the change.
There are some legitimate issues with certain systems that aren’t easily automated today. The issue is with those systems needing to be modernised, but there isn’t a big push for that.
I’d be more concerned as well if this would be an over-night change, but I’d say that the rollout is slow and gradual enough that giving it more time would just lead to more procrastination instead, rather than finding solutions. Particularly for those following the news, which all sysadmins should, the reduction in certificate lifespan over time has been going on for a while now with a clear goal of automation becoming the only viable path forward.
I’ll also go out on a limb and make a guess that a not insignificant amount of people only think that their “special” case can’t be automated. I wouldn’t even be surprised if many of those could be solved by a bog-standard reverse-proxy setup.
Exactly. My “special case” took a little more care, but it works completely fine. Here’s my setup:
I have my router configured to resolve DNS to #2, so I don’t need to hit the WAN to access local services over TLS, and it uses the exact same cert as WAN traffic and the browser is happy.
This is about as exotic as I can think of, and it still works just fine for TLS renewals, and it’s 100% automated. I do need to leave HTTP open (it only serves acme endpoints, so whatever), but I could also close that down and have the renewal process open that temporarily if needed.
The only special case I can think of is a device that rarely turns on, which is incredibly rare these days (you’d generally have an always-on gateway that uses self-signed certs or something for those devices that stay off).
Usually the systems that need to be modernized are working, so nobody wants to invest in a new system that may require retraining the people that may be impacted. Then there’s some systems with integrations that may also require replacing so the integrations can continue to work.
Even then, there’s always a good possibility that the automation fails, especially in the first few iterations of trying to sort out the kinks, and third party automation tools aren’t perfect either. That’s another tool to have to update and maintain once all is said and done.
I’m not trying to rail too hard against the changes, but the impact is especially felt by the people managing the systems, who’s most likely getting more work tacked on to their workload of putting out fires behind the scenes.
I’m not an “actual security researcher” but I was an “actual security officer” at a reeeeally large shop.
Yes, researchers are right. But they don’t dictate what else we have to let slide to allow time to work this constantly.
And neither are they on the hook for it.
They can be pedants, but they can’t do it blind.