Posted on

SolarWinds Hack. Danger Will Robinson!

The SolarWinds hack is worrisome, but probably not for home computer users, although some caution is warranted. This week, the president of Microsoft, Brad Smith, declared the SolarWinds hack was the most sophisticated ever. Before we get to precautions, I’ll explain why the hack is such a big deal.

SolarWinds hack

“I’ve been following this story closely as it has unrolled, and, frankly, it gets worse every day.”

Supply chain hacks

I’ve been following the story closely as it has unrolled, and, frankly, it gets worse every day. It is what the industry calls a “supply chain hack,” an indirect attack on an element in a target’s supply chain.

Instead of striking the target directly— for example, the inventory management system used by the U.S. Treasury Bullion Depository at Fort Knox— the hacker attacks the development facilities of an externally developed product that Treasury uses, an element in the Treasury digital supply chain.

The external product development lab is probably far less protected than Fort Knox. After gaining access, the hackers write in a nasty bit of malware, then wait for the Treasury to install the hacked product. When the product is installed, the hacker has an open door into Fort Knox and can begin dispatching shipments of gold bullion to an off-shore warehouse, Free On Board by the U.S. Army.

I doubt that Fort Knox is vulnerable in the way I’ve described, but a supply chain hack is a method for getting into a highly secure system without confronting the measures put in place by an institution that is guarded like Fort Knox.

SolarWinds hack

SolarWinds, an enterprise software company whose products I once competed with, was an outstanding choice for a supply chain attack. In the last few years, SolarWinds network management system has become popular among Fortune 500 enterprises and government agencies, including the U.S. Treasury.

Network management systems are used to monitor and control computing equipment on a network. Any organization with more than a few dozen computing devices is almost certain to have some sort of network management installed and that system is likely to touch every computer in the organization.

Estimates are that 18,000 companies and government agencies have been infiltrated with vulnerable access points installed in an update to a SolarWinds product.

Personal experience

Personally, I have to think hard about this hack because I could have been a manager responsible for it. I was the technical leader in charge of products similar to SolarWinds. I made many decisions that affected the vulnerability of our products. Could my products have been infiltrated and subverted the same way SolarWinds was caught? I’ve been retired for almost ten years now, so be aware that anything I describe here is likely to have changed.

Nevertheless, I have to say yes. My projects could have been hacked. Quality assurance was a high priority. Some of our best customers were financial institutions and insurance companies who pushed us on security and we increased our security efforts with each release, but portions of our code were written before 2000 when security was not a high priority.

Also, hacking into development often has little connection with engineering. Dishonest, bribed or threatened employees, and rogue contractors all contribute to security vulnerability. Every large organization is bound to have a few bad eggs or weak links.

“In any large public corporation, the stock analysts often hold more sway than the security experts.”

And I must be honest. In any large public corporation, the stock analysts often hold more sway than the security experts. This is one reason I favor products that are certified secure with third party security audits. The best security audits include examination of both engineering and corporate governance, such as hiring procedures and controls on employee integrity. Stock analysts pay more attention to certification, especially certification by prestigious accounting and consulting firms, than opinions from security experts with qualifications a stock analyst probably knows nothing about.

Security at SolarWinds

“A key server is said to have been publicly accessible via a weak password “solarwinds123.”

Unfortunately, there are ample reports that SolarWinds security was poor. A key server is said to have been publicly accessible via a weak password “solarwinds123.” I have wandered computing convention show floors trying passwords like “oracle123” or “goibm” on unattended computers. In the early 2000s, those guesses quit working. Apparently, SolarWinds had some old timers setting passwords. Other poor security practices are said to have been common. Access to SolarWinds servers was also said to be on sale on the dark web.

Ironically, SolarWinds also develops and markets security auditing tools.

Origin of the SolarWinds hack

“I am reminded of the “mole” in author John le Carré’s 1974 spy novel Tinker Tailor Soldier Spy.”

Odds are great that it is a Russian government hack and more likely aimed at espionage and theft of plans and trade secrets than monetary gain. Which is good news for most home users, but the extent of the distribution of SolarWinds transported malware threatens both the U.S. government and economy. The U.S. may be dealing with this breach for years to come. I am reminded of the “mole” in author John le Carré’s 1974 spy novel Tinker Tailor Soldier Spy.

The bad news for home computer users is that criminal hackers may figure out ways to take advantage of the malware installed by the SolarWinds hack to gain access to software installed on home computers.

What to do?

Double down on basic computer security hygiene. I know that hygiene gets tedious, but criminals always go for the weakest victim. A few simple practices go a long way toward making a hack improbable. See my Six Rules for Online Security.

The SolarWinds hack underscores the importance of being careful when downloading and installing new software. Getting your software from established app stores, like the Microsoft Store, Google Play, or the Apple App Store is good practice because the stores vet the software they deliver. You still must be careful: malware has gotten through all of the stores. Software with tons of good reviews that has been downloaded frequently is safest. Never ever download anything from a site that does not show the https locked symbol on your browser. Check the reputation of your vendors and be sure you are on the real site, not a clever spoof.

Also, update your software regularly. Sign up for automatic updates whenever you can. The SolarWinds hack was spread by a software update, but that is not a reason to quit updating. The hack is also being neutralized by automatic updates and will be around far longer if folks neglect updates.

Run anti-malware regularly. The full extent and details of the hack are not yet known, but already anti-malware is cleaning up some of the mess.

Be sensible and be safe.

Posted on

Hacked Phone? Seriously? Yikes!

Someone on social media messaged me asking how to tell if their phone was hacked. Rather than provide a private answer, I’m posting an answer here.

Diagnosing a hacked phone is a complex problem. Ambiguities abound.

Skilled hackers work hard to cover their tracks and you may never know how you were hacked.

Hacked phone symptoms

  • General sluggishness. When you tap or click, the response feels slower than normal.
  • Shortened battery life. If your phone normally goes all day, or several days without recharging, unexpected low power messages midday after normal recharging may be a red flag.
  • Unexpected “ghost typing”, unseen fingers seem to have taken over your keyboard.
  • Unexpected pop-ups.
  • Out going calls you did not make.
  • Ransomware messages, demands for money to regain control of your device. (Other than those from your cell provider!)

Symptoms don’t always mean a hacked phone

In my experience, people think they have been hacked much more often than hacks actually occur because dire warnings of phone hacks catch attention and folks are on edge.

With the exception of ransomware messages, these signs are all ambiguous and each could be more or less innocent. When they occur, think carefully what could be causing them. The first question to ask is: What changed? Did you install a new app? Did you start using an app you have installed but have not used often? Have you changed your habits?

For example, general sluggishness can come from many different sources. If you run short of storage, performance can be affected. Installing an app that squanders resources, or is just too much for your device, can do the same thing. Loading a big batch of photos or videos onto your phone can be also the culprit. Most phones have a storage cleanup utility that may help.

Shortened battery life may be a sign that someone has gotten control of your phone and is using it heavily without your knowledge. However, battery life decreases over time and it may just be old age creeping up on your battery. Or you may have installed an app that is a power hog. Or your habits may have changed.

Ghost typing could be an over-zealous smart keyboard anticipating your thoughts. Or, my own failing, clumsy fat fingers. And for mysterious out-going calls, don’t forget the infamous “pocket dial,” (which occurs much less often with newer phones.) I’ve been fooled into thinking I had a hacked phone when an automatic upgrade kicked in and took over my phone.

Some legitimate apps pop up messages unexpectedly.

Some steps to take

Restarting your phone whenever it acts strange is a good idea. I won’t get into why now, but it often helps. If all is well after a restart, you are probably okay. If your phone is still acting up, try uninstalling anything new. Restart again.

Whenever you suspect you are hacked, try installing and running an anti-malware tool like Malwarebytes or McAfee. Your cell service provider, like T-Mobile or Verizon, may have a free anti-malware tool for you. Phones are less often vulnerable to hacks than other computers because Google and Apple exercise greater control over what you can install on them. I run anti-malware on my phone, but the overhead is high and many of my colleagues prefer not to until they suspect a hacked phone.

If this does not help, the next step is to go to a professional for help. A factory reset is probably on your dance ticket. You can do that yourself, but you may lose stored data, such as photos, contact lists, and stored email and you will probably have to reinstall some apps. Help from a pro can minimize these hassles.

General hack symptoms

Sometimes you are hacked without any of the above symptoms. Skilled hackers work hard to cover their tracks and you may never know how you were hacked. It might have been through your phone, but it could have been through your laptop, even your work computer. Sometimes, you are hacked through a system that you use rather than a computer that you access. Here are some signs that you have been hacked in some way that could have come via a phone hack or somewhere else:

  • Your friends and contacts suddenly get a spate of spam from your email address, indicating that your email has been hacked. The hack could come via your phone or another of your computers. Or it could have been an assault on your email service. (Most of the time, you getting a flood of spam is not a sign that you have been hacked. It’s when your friends complain that you have to worry.)
  • Activity on accounts that you did not initiate. For instance, posts in your name to your Facebook account that you did not post. Worse, credit card or bank account activity that you did not initiate.

The first step is to change the passwords on the bad accounts and contact the account provider. This is especially important for bank and credit card accounts. If you inform your bank or credit card provider promptly, they are required by law to minimize the damage to you. Usually, the bogus transactions will be reversed with no ill effect on you. This is a good reason to review your financial accounts frequently and regularly.

In these cases, I assume that one of my devices have been compromised and look hard for signs of hacking. Then I take steps to clean the computers up, starting with restarts and malware scans. Possibly ending with a reinstall, although that is usually not necessary. In 25 years online, I’ve reinstalled due to hacking only once that I remember. But I’m very careful. If you need professional help, get it.

Final advice

In my experience, people think they have been hacked much more often than hacks actually occur because dire warnings of phone hacks catch attention and folks are on edge. You should on the lookout for hacking, but practicing sound computer security hygiene, the chances you will be victimized go way down, especially if you are not a public figure with a target on your back. Cybercrime is more prevalent than ever before, but the victims are most often deep-pocketed businesses and public figures. Check out my six rules for online security.

Posted on

Free Digital Services: TANSTAAFL? Not Exactly

For the last two decades, we’ve been in an era of free services: internet searches, social media, email, storage. Free, free, free. My parsimonious head spins thinking of all that free stuff.

Social and technical tectonic plates are moving. New continents are forming, old continents are breaking up. Driven by the pandemic and our tumultuous politics, we all sense that our world is changing. 2025 will not return to 2015. Free services will change. In this post, I explain that one reason free digital services have been possible is they are cheap to provide.

I’m tempted to bring up Robert Heinlein, the science fiction author, and his motto, TANSTAAFL (There Ain’t No Such Thing As A Free Lunch), as a principle behind dwindling free services, but his motto does not exactly apply.

Heinlein’s point was that free sandwiches at a saloon are not free lunches because customers pay for the sandwiches in the price of the drinks. This is not exactly the case with digital free lunches. Digital services can be free because they cost so little, they appear to be free. Yes, you pay for them, but you pay so little they may as well be free.

A sandwich sitting beside a glass of beer costs the tavern as much as a sandwich in a diner, but a service delivered digitally costs far less than a physical service. Digital services are free like dirt, which also appears to be free, but it looks free because it is abundant and easily obtained, not because we are surreptitiously charged for it.

Each physical service sale has a fixed cost for reproduction and delivery. Digital services have almost no cost per sale for reproduction and delivery, until they have invest in expanded infrastructure.

Free open source software and digital services

One factor behind free digital services is the free software movement which began in the mid -1980s with the formation of the Free Software Foundation. The “free” in Free Software Foundation refers more to software that can be freely modified rather than to products freely given away, but most software under the Free Software Foundation and related organizations is available without charge. Much open source software is written by paid programmers, and is therefore paid for by someone, but not in ways that TANSTAAFL might predict.

The workhorse utilities of the internet are open source. Most internet traffic is Hyper Text Transmission Protocol (HTTP) packets transmitted and received by HTTP server software. The majority of this traffic is driven by open source Apache HTTP servers running on open source Linux operating systems. Anyone can get a free copy of Linux or Apache. Quite often, servers in data centers run only open source software. (Support for the software is not free, but that’s another story.)

So who pays for developing and updating these utilities? Some of the code is volunteer written, but a large share is developed on company time by programmers and administrators working for corporations like IBM, Microsoft, Google, Facebook, Amazon, and Cisco, coordinated by foundations supported by these same corporations. The list of contributors is long. Most large, and many small, computer companies contribute to open source software utilities.

Why? Because they have determined that their best interest is to develop these utilities through foundations like the Linux Foundation, The Apache Foundation, Eclipse Foundation, Free Software Foundation, and others, instead of building their own competing utilities.

By collaborating, they avoid wasting money when each company researches and writes similar code or places themselves in a vulnerable business position by using utilities built by present or potential competitors. It’s a little like the interstate highway system. Trucking companies don’t build their own highways, and, with the exception of Microsoft, computer companies don’t build their own operating systems.

Digital services are cheap

Digital services are cheap, cheaper than most people imagine. Free utilities contribute to the cheapness, but more because they contribute to the efficiency of service delivery. The companies that use the open source utilities the most pay for them in contributed code and expertise that amounts to millions of dollars per year, but they make it all back because the investment gives them an efficient standardized infrastructure which they use to build and deliver services that compete on user visible features that yield market advantages. They don’t have to compete on utilities that their customers only care about when they fail.

Value and cost of digital services are disconnected

We usually think that the cost of a service is in some way related to the value of the service. Digital services break that relationship.

Often, digital services deliver enormous value at minuscule cost, much lower cost than comparable physical services. Consider physical Encyclopedia Britannica versus digital Wikipedia, two products which offer similar value and functionality. A paper copy of Britannica could be obtained for about $1400 in 1998 ($2260 in 2021 dollars). Today, Wikipedia is a pay-what-you-want service, suggested contribution around $5 a month, but only a small percentage of Wikipedia users actually donate into project.

Therefore, the average cost for using Wikipedia is microscopic compared to a paper Britannica. You can argue that Encyclopedia Britannica has higher quality information than Wikipedia (although that point is disputable) but you have to admit that the Wikipedia service delivers a convenient and comparable service whose value is not at all proportional to its price.

Digital reproduction and distribution is cheap

The cost of digital services behaves differently than the cost of physical goods and services we are accustomed to. Compare a digital national news service to a physical national newspaper.

The up-front costs of acquiring, composing, and editing news reports are the same for both, but after a news item is ready for consumption, the cost differs enormously. A physical newspaper must be printed. Ink and paper must be bought. The printed items must be loaded on trucks, then transferred to local delivery, and hand carried to readers’ doorsteps. Often printed material has to be stored in warehouses waiting for delivery. The actual cost of physical manufacture and delivery varies widely based on scale, the delivery area, and operational efficiency, but there is a clear and substantial cost for each item delivered to a reader.

A digital news item is available on readers’ computing devices within milliseconds of the editor’s keystroke of approval. Even if the item is scheduled for later delivery, the process is entirely automatic from then on. No manpower costs, no materials cost, minuscule network delivery costs.

The reproduction and network delivery part of the cost of an instance of service is often too small to be worth the trouble to calculate.

My experience with network delivery of software is that the cost of reproducing and delivering a single instance of a product is so low, the finance people don’t want to bother to calculate it. They prefer to lump it into corporate overhead, which is spread across products and is the same whether one item or a million items are delivered. The cost is also the same whether the item is delivered across the hall or across the globe. Physical items are often sit in expensive rented warehouses after they are reproduced. Digital products are reproduced on demand and never stored in bulk.

Stepwise network costs

Network costs are usually stepwise, not linear, and not easily allocated to the number of customers served. Stepwise costs means that when the network infrastructure is built to deliver news items to 1000 simultaneous users, the cost stays the about same for one or 1000 users because much of the cost is a capital investment. Unused capacity costs about the same as used capacity— the capacity for the 1000th user must be paid for even though 1000th user will not sign on for months.

The 1001st user will cost a great deal, but after paying out to scale the system up to, say, 10,000 users, costs won’t rise much until the 10,001st user signs on, at which point another network investment is required. Typically, the cost increment for each step gets proportionally less as efficiencies of scale kick in.

After a level of infrastructure is implemented and paid for, increasing readership of digital news service does not increase costs until the next step is encountered. Consequently, the service can add readers for free without affecting adversely affecting profits or cash flow, although each free reader brings them closer to a doomsday when they have to invest to expand capacity.

Compare this to a physical newspaper. As volume increases, the cost per reader goes down as efficiency increases, but unlike a digital service, each and every new reader increases the overall cost of the service as the cost of manufacturing and distribution increases with each added reader. The rate of overall increase may go done as scale increases, but the increment is always there.

Combine the fundamental low cost of digital reproduction and distribution with the stepwise nature of digital infrastructure costs and digital service operators can offer free services much easily and more profitably than physical service providers.

The future

I predict enormous changes in free services in approaching months and years, but I don’t expect a TANSTAAFL day of reckoning because providing digital services is fundamentally much cheaper than physical services.

I have intentionally not discussed the ways in which service providers get revenues from their services, although I suspect most readers have thought about revenue as they read. The point here is that digital services are surprisingly cheap and their economic dynamics are not at all like that of physical goods and services.

As digital continents collide, I expect significant changes in free digital services, changes that will come mainly from the revenue side and our evolving attitudes toward privacy and fairness in commerce. These are a subjects for future posts.

Posted on

How the Network Works: Routing

In a comment on a previous post, Steve Stroh suggested explaining the nuances of routable and non-routable addresses. This distinction is important for home network security, but without a little background in computer networking, the concept doesn’t mean much. This post explains a little about how the global computer network operates.

Traditional telephone service

I’ll begin with traditional telephone service works because, without realizing it, most people have it tucked away somewhere in their brain that computer networks work like an old telephone system.

Circuits

Traditional telephones are based on circuits. Imagine an old-fashioned switch board with a bunch of incoming wires and outgoing sockets. When an incoming wire from your phone is plugged into an outgoing socket a circuit is completed and you can speak to and hear a person on a telephone at the other end of the circuit.

In a simple time, when Fred and Ethel were struggling performers, they both roomed at The Algonquin. Its switch board could manage the handful of telephones in the hotel. Fred could pick up his phone, tell the switch board operator to connect him to Ethel, a circuit between Fred’s phone and Ethel’s phone was made and Fred and Ethel could plan, dream, and toss whoopie.

When Ethel’s fan dance made headlines, she moved to The Ritz. Fred, who was a baggy pants comic with a fake Yiddish accent, was stuck at the Algonquin. For Fred to call Ethel, the Algonquin operator had to connect to the Ritz operator, who then connected Fred’s line into Ethel’s phone. Unfortunately, Ethel no longer talked to lowlifes like Fred and she immediately hung up. The operators unplugged the lines, and the circuit was gone.

The pattern of connecting switch board to switch board was repeated until the phone network covered the entire U.S. and transoceanic cables extended the network to Europe. Getting a connection could take hours, but the system worked. Over time it was automated, first by mechanical relays, later by transistors and computer chips. Connections became faster. Nonetheless, for most purposes, the circuit system was eventually abandoned in all but metaphor.

Packet switching

Computer networks replaced circuits with an older approach: mail packets.

Mail works differently than the telephone. Writing an address on an envelope is only superficially similar to dialing or asking an operator for a phone number. When you drop an envelope in the mail, you have a promise that the post office will try to deliver it, but that promise is no guarantee and the office that first receives the letter is likely not to have any knowledge of the letter’s destination.

A mailbox will accept a letter addressed to T. H. E. Wiz, 1 Emerald Way, Oz, Kansas without a murmur, although neither T.H.E Wiz, Emerald Way, nor Oz exists in Kansas. You may or may not find out later your letter was undeliverable. It will rattle around the postal system until it is eventually returned or falls into a dead letter bin.

On the other hand, try dialing a stage number, like The Bionic Woman’s number, 311-555-2368. You are told that a connection is impossible as soon as you finish dialing.

The crucial difference is that before your message is transmitted by phone, i.e. you begin to speak, the path you will use to communicate is either is made or fails. The postal system, on the other hand, accepts your message, then passes it on until it lands in an office that recognizes the address and the intended recipient.

Circuits v. packets

Off-hand, the telephone seems like the smarter way. Why go to all the trouble of shipping an envelope around the country when you can decide before sending the message if delivery is possible? Isn’t the post office method a step backward?

Well, no. Circuits may seem more efficient, but they don’t fit well into the reality of the global internet, which is huge, ever changing, and implemented in a patchwork of wildly varying speed and reliability; a salmagundi of large and small, public and private entities that is closer to a frontier pony express than an orderly telephone system.

No single entity understands the complexity of the global network. However, piecing together a workable, let alone optimal, circuit requires just that.

A typical computer message is broken up into a series of independent small packets, each with its own address. These are dumped willy-nilly into the network and each is passed from router (the computer equivalent of a switchboard or a post office) to router until they arrive at their destination and are reassembled. Some will be duplicated to be sent on alternate routes or further broken down into smaller packets to optimize transmission on equipment that can’t handle large packets. The process is messy, but resilient and makes good use of available resources.

Packets in the network hop from router to router approaching light speeds, i.e. almost instantaneously, but then they sit in a buffer while the router decides what to do with it. A packet can hop to another router faster than a router can respond to a query on traffic conditions, so why bother asking? Sending packets to find their own way only requires local knowledge of the condition of the communications infrastructure. With a network as extensive and varied as the global computer network, this is a critical advantage.

Today, most telephone service, including cellphones, travels in switched packets that simulate circuits. We still tend to think of a single wire connection that runs from one phone to the next through a myriad of automated switch boards, but in fact most of the time, our voice is carried in packets drifting through a global network.

Permanent (static) addresses

In the old days, computer addresses, IP addresses, were more or less permanent. Businesses requested and were assigned blocks of addresses. A system administrator had to keep track of those addresses and dole out new addresses from the list as computers were added to the network and return old addresses to the available list as computers were decommissioned.

It was an exacting job. If the list got scrambled and two devices used the same address, the network would behave erratically. Small businesses typically had only a few addresses, which limited the number of computers they could use.

Private non-routable addresses

Giving every computer a public IP address was also a security problem. Each of those publicly addressed computers were vulnerable to direct outside attacks. They had to be managed carefully to prevent intrusion. Individual users seldom had the training and temperament to do that job well.

A solution from the mid-1990s was to declare blocks of addresses private or non-routable. The largest private block (10.0.0.0 – 10.255.255.255) has over 17 million individual addresses. The second block (172.16.0.0 – 172.31.255.255) over a million addresses. The smallest block (192.168.0.0 – 192.168.255.255) has over 65 thousand addresses.

This had lots of advantages. System administrators could devise address assignment schemes for the private address blocks that were relatively easy to manage with subsets for buildings, floors, departments, etc. and not worry about clashing with other businesses. Since the addresses were non-routable, computers with private addresses were easier to isolate from intruders.

Dynamic Host Control Protocol (DHCP)

The process of assigning and re-assigning IP addresses was automated with Dynamic Host Control Protocol (DHCP). When a computer connects to a home network, the home router, following DHCP rules, assigns the computer an IP address from one of the private blocks. Which block depends on how DHCP is implemented on the router.

When an Internet service provider connects a home router to the global network, the service provider uses DHCP rules to assign a unique public and global IP address to the home router.

Network Address Translation (NAT)

From the outside, each home network looks like a single computer with one IP address. This presents a problem: how to connect those internal private addresses with the single outward-facing address. Non-routable private IP addresses combine with DHCP and Network Address Translation (NAT) to solve this problem. Network Address Translation (NAT) is a protocol implemented on routers that handles coordinating internal non-routable IP addresses with the single external public IP address that the router presents to the global network.

Without private addresses, DHCP, and NAT, connecting a new computer to a home network would be a tricky and exacting job. I won’t say that connecting a new device is easy today, but I assure you, when I think about managing networks thirty years ago, I am amazed. Our grandsons have our wireless network id and password stored on their phones and laptops. When they walk into our house, their laptop or phone connects with our wireless network, an IP address is assigned, and they connect with the global network without me doing anything. Astounding!