In March 2022, IoT security company Armis released details about three security vulnerabilities in APC UPSs that they collectively named "TLStorm". They pitched TLStorm as a massive security disaster for the 20 million APC UPSs out there, and many tech media sites and other parties have re-reported their findings. Armis themselves say on this page that attacks on UPS devices involving "explosions" are "no longer fictional". You'd be forgiven for thinking that someone can just hack in from anywhere, and set any UPS on fire.
I have a different take on this situation, informed by many years of using, fixing and reverse engineering these devices, and I'm going to share that here. The below is just an opinion, and it's an opinion from a hardware engineer not a security analyst - so if you have a problem with my blog, go read someone else's blog.
I'm unimpressed, but unsurprised, by the way TLStorm has been marketed, and this hype re-posted by many who don't actually have the knowledge necessary to fact-check it and assess its real significance. The reality is less exciting than people think, and generally much less exciting. Let me explain.
TLStorm is a collection of three vulnerabilities in APC Smart-UPSs. Only with at least two is there a way for an attacker to gain entry to a system, and then perform a hardware attack on a UPS. Yes, in that case, with enough effort, attackers can destroy a UPS and potentially damage connected devices as well. However, only APC's SmartConnect Smart-UPSs are (or were) affected by all three vulnerabilities, and they make up a small minority of all the APC UPSs out there. Think well under a million units.
There are a greater number of UPSs - I'd estimate maybe a few million at most - that are only affected by one of the vulnerabilities, CVE-2022-0715. This vulnerability affects all Smart-UPS products sold since about 2009. The 20 million includes many UPSs sold before 2009 though, and many other product lines - none of APC's consumer product lines are even affected by this, for example. So it's quite misleading.
Besides the misleading 20 million unit claim, on the page I linked above Armis clearly imply that all of these units (the ones affected by CVE-2022-0715) are insecure. They make quite a big deal out of it too, claiming it represents a significant security risk to organisations. But how bad is CVE-2022-0715 on its own? Are all Smart-UPSs sold since 2009 really insecure, outside of just the SmartConnect ones?
Firstly I'll mention that I basically knew about CVE-2022-0715 roughly 5 years before Armis published the TLStorm disclosure. I just didn't take the "CVE" approach. CVE-2022-0715 affects the UPSs I have, and some of my dealings with them have effectively taken advantage of part of this vulnerability. That's how I knew about it well in advance of it becoming public knowledge. CVE-2022-0715 isn't fixed on pre-SmartConnect UPSs yet - and whether a fix is or isn't going to be released may depend on whether these UPSs have the hardware capabilities required to implement the type of firmware security that Armis is expecting. But I, for one, am not in a hurry for it. My point is to actually question this hype, and also to mention a bad side effect of going too far with embedded security.
To me, calling CVE-2022-0715 a vulnerability is a bit like like calling your car unsafe because you could put the wrong fuel in the tank and damage it. The point is that you generally don't do that by accident, and you also expect that any nefarious actors won't have access to your fuel tank in the first place so it won't really matter (because you've locked your car - you have valuables inside which you care about as well). Likewise with UPSs, CVE-2022-0715 relates to the way that the UPSs don't check that a firmware image came from APC before installing it. But would-be hackers should never have access to upload a firmware image in the first place, so it shouldn't even matter. And if they do have that access because of poor security practices and the like - then they can already shut down the UPS and mess up your network without making use of CVE-2022-0715. This vulnerability doesn't let you remotely take over any UPS, it simply means that you can do more with that UPS if you've already got access to its interfaces. This is not the security disaster that it's being made out to be.
APC symmetrically encrypts their firmware files in 8 byte blocks, decrypts them on-chip during the device firmware update (DFU) process, and generally doesn't enable readout protection on microcontrollers in their products. The vulnerability states that "if a key is leaked" the key could be used to correctly encrypt malicious firmware, which could then be uploaded to the UPS. But saying "if a key is leaked" actually downplays the situation. Since APC doesn't enable readout protection, the code that does the decryption, and thus the keys themselves, are available on every single UPS out there. Heck, I've got the files myself, for several models - and I could send them anywhere. The point is the key is already leaked, it's all over the place if you know how to reverse engineer firmware, and it has been since 2009 when the products were first released. It sounds so terrible, such an ideal way to destroy an organisation, to hack in and upload malicious firmware to make their UPSs explode. And yet we have no documented cases of this "vulerability" ever being utilised, in the 13 years that it's existed. Underwhelming, isn't it?
As I said, to make use of this vulnerability you need some sort of access to a UPS to upload firmware in the first place - and attackers don't simply get that by magic. On non-SmartConnect UPSs - again, the majority of them - it can be done by two methods:
If you have phyical access to the device, you can plug in a USB or serial cable, or even slot in a network management card. But if you have physical access to the device, there are much easier ways to cause trouble. It's like with the car - if you have the keys to someone's car and want to cause trouble... putting diesel in the tank is probably not the first thing you'd think of.
With that said, it's also possible remotely in some circumstances. The firmware can be updated via a network management card, but this is easily password protected, and no vulnerabilities were found in APC's network management cards as part of TLStorm. So I don't see this potential avenue as a serious problem... for anyone. Just put a decent password on your network management card. Stop expecting the device to cover for users' mistakes.
You could also do it if you had admin access to a PC that was already connected to the UPS by USB or serial. To me, this is probably the main way that this vulnerability is likely to be used, and it's still really not very practical. For a start, these UPSs are generally not monitored that way in the sorts of IT environments that hackers would like to target - network management cards are a better solution. And even if you do have a UPS being monitored by a PC over USB or serial, that PC will generally be powered by the UPS itself. On non-SmartConnect UPSs, the DFU process turns the UPS's output off during a firmware upgrade, and does so before the firmware has even begun to upload. I don't care how good of a hacker you are - you need to load the firmware using APC's bootloader in the first instance, and it shuts the UPS down before doing so. There's no way to cache your firmware virus on the device without turning off the output first - I've seen the hardware and used the bootloaders, I can guarantee that. So if the compromised PC you're doing this from is powered by the UPS, as it will be in most cases, all you will achieve is to shut down the system in question, and at best potentially wipe the UPS's firmware leaving it running only the bootloader, so it can't be used until the firmware update process is re-run. Either way, it's not hardware damage.
And if you've got that kind of access to a PC on some network, it would be far easier to just upload ransomware and destroy things at a software level, than to conduct a complicated, ad-hoc engineering attempt to write custom firmware for a UPS to attempt to destroy hardware. It's just an impractical vulnerability to make any use of in the vast majority of situations - and very labour intensive compared to the alternatives you'd already have if you had physical access to the UPS or remote access to a system connected to it.
Even aside from all the above hurdles, writing custom UPS firmware to exploit this would take a serious amount of time and knowledge of a product, and the firmware required will be substantially different for each model of UPS. Imagine being a hacker and being 99% ready to deploy your amazing insta-explode UPS firmware, when you realise that your target actually has a SMX1000I, not the SMT1000I you've written firmware for! Oh no - now you've got to spend another 3 months learning code for a different microcontroller and figuring out how to write firmware that even runs on that microcontroller. And yes, if you're wondering - the SMT1000I and SMX1000I do have different microcontrollers, and completely different hardware architectures, meaning the potential options for wreaking havoc will differ. That's also not to mention testing it - you'd need your own UPS of the same model you're trying to hack for a start, and there are a multitude of models and generations within the non-SmartConnect units affected by CVE-2022-0715.
It's a hard job writing power control firmware that works and does what it's supposed to do, be that proper operation or proper catastrophic hardware damage - and anyone with the skills and patience to do it would be wise to consider switching careers to become an electronics engineer. Just saying.
Armis has used the words "ignite" and "explode" to describe what one can do with malicious firmware - but the reality is that Smart-UPSs are all metal cased units with hardware protection devices like battery and mains fuses, and I doubt one could actually start a fire from software on any of the lead-acid battery based units. Overcharging the batteries by software is difficult on many units - for example, the battery chargers in the SMT series simply aren't capable of exceeding safe charging voltages in my experience, so the worst you could do is very slowly cook someone's batteries over weeks or months, by setting it above the recommended voltage. That's not very exciting - and these UPSs also have fans and/or passive ventilation to disperse any hydrogen gas generated by the batteries as well. UPSs with lithium battery technologies might be a different story in this regard, but only if the BMS is controlled by a microcontroller for which firmware updates are possible - not at all guaranteed, and the vast majority of UPSs have lead acid batteries anyway.
Armis did an impressive demonstration where they made a SmartConnect SMT series UPS release a lot of smoke. But there is a big difference between releasing smoke by destroying capacitors and MOSFETS, and creating a genuine fire hazard. As well, the methods they used to produce that smoke, and the results available, will depend on the model and series of UPS, because they each use different hardware architectures and control methods.
In terms of damaging connected devices rather than the UPS itself, Armis generated some impressive overvoltages with their SMT series UPS - as I have accidentally done several times while working on similar units by damaging inverter control circuitry! However, the UPSs have hardware protection like MOVs which may wreak havoc with such attempts, and many connected devices (for example computers with switching power supplies) won't be susceptible to minor overvoltages anyway. What I mean is that, of course your 230V PC is not designed to operate reliably on 350V, but you may need to go higher for something immediate and exciting to happen - often power supplies will tolerate significant abuse temporarily, and then fail matter-of-factly rather than by catching fire. And the amount of overvoltage actually achievable from a UPS by software will vary depending on the UPS in question. For the SMX series for example, creating significant output overvoltages (beyond about 350V RMS) won't be possible because the output voltage is derived from a boost converter that is not software controlled in the first place.
Don't take this as criticism of Armis, their motivations or their results - their hacking and destruction of a SmartConnect SMT rackmount UPS was extremely impressive, and I feel vindicated seeing the SmartConnect system go up in flames. I never liked it anyway. And the world does need people to test these systems and point out security flaws. But I just think that CVE-2022-0715 as a standalone vulnerability, and the significance of this for non-SmartConnect UPSs, is being overplayed here. This means that the size, and potential impact, of TLStorm as a whole is being overplayed, because most UPSs still aren't SmartConnect. So what I do kind of critisise is the marketing aspect of this disclosure. Owners of non-SmartConnect UPSs can easily make sure their systems are safe by adequately protecting the network management cards or PCs that they have connected to their UPSs. And they don't need firmware upgrades to mitigate anything.
Why do I care about this? Because as a hardware engineer, but also a longtime hobbyist and repairer, I'd rather have devices where the firmware is readable and isn't locked crazily at a hardware level in the name of security. Fixing CVE-2022-0715 shouldn't require readout protection on any of the internal UPS microcontrollers - you only need to verify a firmware image using a public key on the device, so there's nothing that needs protection in the name of security. But it's likely that readout protection will be employed by Schneider Electric and other brands in the wake of this bad publicity on security, because it helps keep those prying security researchers from causing trouble. And that's a sad thing. I fixed the UPS mentioned here because I was able to read firmware from an identical unit and copy it to the broken unit, and also repaired an APC SMX (Smart UPS X - extended run) series UPS which had a failing microcontroller, where I was able to read the firmware and copy it to a new microcontroller. Clearly, APC didn't enable readout protection for the sake of their intellectual property in these cases, and you can imagine why: what is someone going to do with an APC UPS firmware binary, except run it on an APC UPS?
Personally, I'm fine with just protecting the outside of the device from attack like I already do, rather than by locking the device itself internally, which is why I didn't even lift a finger when the vulnerabilities were disclosed. It meant nothing to me despite the fact that I use five affected UPSs, because like for a lot of bigger system administrators, I use network management cards which are password protected - that's not trivial to get into, and if someone does hack in that far, they can already shut down the UPS, and I have bigger things to worry about than them messing with the firmware. And I can see why it's good to implement firmware signing in addition to those external protectons, and why enterprise customers would want that a lot more than me, but CVE-2022-0715 is just not the big deal it's being portrayed as here, and the unfortunate, likely side effect of this whole debacle is that in the future, you won't be able to do the same sort of repairs on these devices, because everything will be read protected.
And I know people are going to say that almost no one repairs these things anyway, so it doesn't matter. But unnecessarily locking down things is just another thing in a long list of things that make our technology inherently less accessible and worse for the environment, because it becomes unserviceable without manufacturer support. It's not the right thing to do.
It doesn't seem impossible for APC to fix CVE-2022-0715 on their existing UPS models affected by it, as long as it doesn't require more capable microcontrollers to support the extra code required to implement firmware signing. But a lot of these products haven't had firmware updates released in years, and some use quite old microcontrollers and different combinations of technologies which may require different approaches for different products - it sounds like a lot of work, and I suspect they'll be reluctant to do it, especially with the fact that CVE-2022-0715 on its own isn't a practical issue to most people. But it will be interesting to see what they do. I'll have a look and cover it in more detail here if they do release something - and of course I'll be keeping copies of all the old firmware binaries in case I don't like it!
Firmware updates have been released for most units to mitigate the exploits. Some updates prevent firmware installation via the NMC, with Schneider Electric's latest security notification still saying that "a future firmware update will be released to re-enable this feature", which hasn't happened.
However for older SMT, SMC, SMX and SRT series UPSs, we have this wonderful solution, from the company labelled the "World’s Most Sustainable Corporation 2021" by the Corporate Knights Global 100 Most Sustainable Corporations list...
"UPS models from these series with the specified IDs have been phased out and firmware remediation is not available for them. To remediate the vulnerabilities, we recommend that you replace UPS models with the specified IDs with a newer version of a similar model."
In other words: throw it out and buy a new one. A new unit of the same model, with the same capabilities. Even if the old one still works. Just to fix this firmware issue.
I rest my case.
None yet!
Display Name
Email (optional, not displayed)
Website (optional, displayed, include "https://")
Comment (up to 1000 characters)