I live 4 miles from one oil refinery and 6 miles from another. I don’t think about it much. But I did this morning when I read an article in the MIT Technology Review on the Triton malware.
I don’t know much about the two refineries. Forty years ago, when I was a carpenter I worked for a few months on a construction project inside one of them. That was probably the safest construction site I ever worked on. There, we followed safety rules unheard of on other sites. Forget to snap on your safety rope doing high work and somebody would yell at you before you got your hammer out of your belt.
And the rules had teeth. As I remember, intentionally break a safety rule and you were outside the gate, which was not trivial. Industrial carpenter work paid way more than residential or commercial work and double-time overtime was regular. I think I got triple-time for working overtime on a Fourth of July.
That was in the early 80s when OSHA work rules were a cat and mouse game construction workers played for fun. If I had followed OSHA rules regularly then, instead of trying to avoid them, or landed more refinery jobs where they were serious about the rules, I probably wouldn’t wear a hearing device today and fewer of my brother carpenters would be missing body parts or have died from asbestos cancer.
So. I have not thought much about safety in the refineries. I’ve seen my share of sloppy computer security in industrial plants of in past years: default passwords, work stations left unlocked without attendants, and unpatched or outdated software were all common in plants where products I worked on were installed, but I never connected the dots to the refineries next door, which I thought of as paragons of safety.
I did this morning.
Triton is malware that was discovered in a refinery in Saudi Arabia. Hackers breached plant computer security in 2014 and began to infiltrate the system with the Triton malware. They caused a plant shutdown in June 2017, which raised suspicions but did not lead to detection. A second shutdown in August 2017 revealed the Triton attack. Neither of the shutdowns caused more than inconvenience, but the subsequent investigation revealed that the hackers were prepared to wreak massive damages and deaths.
To explain how Triton works, I have to explain how modern industrial control works today. In some ways, industrial plants are much safer than they were when I was a carpenter. Something called a SCADA (Supervisory Control And Data Acquisition) controls many industrial processes today.
One summer during college, I was SCADA in a pea freezing plant. They called me a “tunnel man.” My job was to walk a circuit in a refrigerated freezing belt tunnel, checking temperatures, salinity levels, and progress over the belts. I reported to the refrigeration engineers several times an hour with the measurements. If something jammed the belt, I had to sound the alarm so that the engineers could turn down the refrigeration units so the belt would not freeze up solid.
In those days, I was fairly responsible, and I took my job seriously. I realized that if I messed up, the damage could be great. However, the truth was that the job was totally mindless and boring after the first week. A trained pigeon could have done the job at least as well as I did.
The old pea freezing plant closed decades ago, but today the tunnel man would be replaced by SCADA. Sensors would relay temperatures and salinity to a control dashboard, other devices would detect issues on the belt, and most of the control would be automated. A SCADA system monitors continuously instead of my periodic inspections and react quicker and more precisely than the engineers listening to some kid inaccurately describe ice buildup on the Kelly belt. Freezing plants may still have tunnel men as backups, although with labor costs today, I doubt it.
SCADA systems are not perfect, but they are much better than humans or trained pigeons for mindless relay of information and rote response. Each year SCADA, with the help of advancing sensing and control algorithms, gets better. But suppose, someone tampers with the sensors? Or the control rules? Even in an innocuous vegetable freezing plant, an exploding ammonia tank could be quite dangerous.
Hence the need for security. Current industrial computer security is built in layers. A plant computer system is almost certainly connected to the internet. But good security practice is to divide a plant network into several layers and segments. One layer is connected to the internet behind a firewall like any good business security set up. Inside that perimeter, the SCADA system subject to further security controls and is less accessible. Within SCADA, there are often other segments that are further isolated, usually to the point of “air-gapping,” complete physical separation from other computing equipment. This level of security is usually reserved for critical emergency controls that keep the most dangerous processes within safe limits. In theory, these systems are untouchable.
Now the part that made my blood run cold: Triton delivered control of air-gapped critical safety controls to outside hackers.
One of the truths of modern computing is that air-gapped systems can be penetrated. Essentially, the attacker infects the surrounding systems with software that lies in wait, looking for connections to critical hardware controllers, and pounces when it detects a connection. Without the most stringently enforced human security, eventually some hapless technician connects an infected laptop or similar device to the air-gapped system “just for a minute” and the critical system is compromised. Using this technique, the US and Israel compromised and brought down Iranian uranium centrifuges in 2010. Russians brought down Ukrainian power-plants in 2015.
There is always some uncertainty in tracing this kind of hack, but best current opinion is that control of the Saudi refinery was in the hands of a government industrial institute in Moscow for a period in 2017.
Triton appears to have been neutralized. The controller that Triton targeted has been patched. Security practices at the Saudi plant have been revamped. If you are curious, you can read about many of the details, even Python code for detecting Triton here.
I am not likely to purchase industrial gas masks soon. Homeland Security has been helping critical industries to harden their processes (check it out here) and the US still attracts the best computer engineers from everywhere on the planet.
But this is no time to be complacent. Frankly, I have not been impressed with the sophistication of our government in cybersecurity, but I do everything I can to encourage them to do more. The Russians, Chinese, and North Koreans have invested heavily in cyber-warfare. I’ve sent letters to my congressional delegation urging them to fund support for cybersecurity in general and industrial cybersecurity in particular. I urge you to do so also.
One Reply to “My Blood Ran Cold This Morning”