Supply-Chain Hacks and Hardware’s Role in Security

Over the past years we’ve seen companies rush to secure the expanding reach of cloud and software deployments. And rightfully so, as recent F5 Labs research shows applications as the initial targets for over half of reported breaches. But as companies hasten to shore up this side of the business, they need to remain vigilant and keep an alert watch on what’s happening on the infrastructure and hardware side, too.

The recent Bloomberg article is not the first claim to be made regarding what’s come to be known as a supply-chain hack. Reports of third parties intercepting networking equipment in the supply chain for the purpose of tampering with, spying on, or otherwise compromising data have noticeably floated around the Internet for the last five years at least.

Indeed, the threat of tampering is not constrained to the hardware supply chain. It’s a straightforward enough way to slip into the process, but it is not the only entry point. To be clear, there is no part of a system today—from hardware to firmware to OS and software—that is not subject to some kind of threat. Operating system and software compromises are seen in articles describing malware and exploited vulnerabilities every day. Persistent threats have been delivered via BIOS (firmware) rootkits. And the increasing use of third-party manufactured subsystems over the past decade lends credence to the feasibility of a supply-chain hack impacting board-level hardware. 

Yes, it is possible. Yes, a supply-chain attack would be difficult to detect. And yes, as long as hardware plays a role in delivering applications in data centers and cloud environments (read: for the foreseeable future), it’s something organizations need to think about—whether or not specific vendors rely on their own hardware or someone else’s.

Secure Systems Need to Take Hardware into Account

Hardware and firmware-based attacks pose a unique risk compared to other types of attacks, which in part explains the deep level of concern organizations have on the topic.

Hardware and firmware-based attacks are attractive to bad actors because of:

1. Low detectability. Firmware operates in the background and is often not visible to software. Modern systems discourage access to even the BIOS firmware, as inappropriate changes can render a system unusable.
2. High persistence. Once established, it is difficult to rid an infected platform of the malware. While operating systems can be reinstalled with relative ease, restoring firmware to its original state is a much more difficult proposition.
3. High privilege. Any code executing in hardware or firmware has direct hardware-level access. Malware with this level of privileges can spy on and even control the entire system.

Add it all up, and there are tangible advantages of companies that maintain control over their own hardware. For instance, F5 oversees each step—from design, to prototype, to fabrication, and finally manufacturing—to ensure the integrity of our systems. Without this level of control, vendors can offer comparatively limited assurance that the hardware supporting their systems is protected.

The Challenge of Securing What You Don’t Control

At this point, I’ll mention that F5 does not maintain a manufacturing relationship with Super Micro or any other server motherboard OEM. (For more details, please reference the AskF5 knowledge base article related to the recent industry news.) This moment does provide a good opportunity, though, to reinforce the significance of companies that can employ a rigid set of processes to protect against tampering with its hardware, firmware, and software at any point during development, manufacturing, and assembly. After all, if you have to rely on hardware somewhere along the line, doesn’t it make more sense to rely on a vendor with hardware expertise?

While you’ll undoubtedly find vendors that are all too willing to make the hardware elements of security “someone else’s problem,” this approach is often incomplete. If you need assurances around the hardware that is supporting your infrastructure, you want a vendor that isn’t going to play Hot Potato with the responsibility of ultimately protecting your assets. So, for the remainder of this article, we’ll explain what F5 does to mitigate this risk for its customers, and then identify some questions you can ask of vendors to make sure they are responsibly limiting your exposure. But here’s the tl;dr version: If you don’t know how or where your infrastructure hardware is designed, manufactured, tested and assembled, you might have reason to be concerned—with F5, on the other hand, the process never leaves our oversight.

What F5 Brings to the Table: The Complicated Bit

F5 is headquartered in Seattle and our hardware design and development takes place at a company facility east of Seattle in Spokane. This team oversees every aspect of F5 hardware development. Born out of a desire to develop hardware specifically to power our ADC platform, BIG-IP, this allows us to integrate fanatical attention (in a good way!) to the security of our hardware.

It helps to pause here for a (very) quick overview of how hardware is designed and developed. The preliminary design is done in CAD software, from which a vector image of the board is generated. This is called a Gerber file, based on the de facto industry standard format for these images. From that file, a printed circuit board (PCB) is fabricated. The PCB is just the base board—the circuit traces and pinouts. The next step is to fabricate a printed circuit assembly (PCA), which adds the actual components (CPUs, memory, ICs, transistors, etc.) to the PCB.

F5’s processes also include post-manufacturing testing to aid in protecting against a supply-chain hack from third-party suppliers. We use a combination of AOI (Automated Optical Inspection) as well as 5DX or AXI (X-ray) inspection. Each are designed to find a variety of issues that impact quality and the integrity of the system, including identifying any element that isn’t part of the product design.

Both AOI and X-ray inspection are automated processes that begin with Gerber files that describe in painstaking detail exactly how the board should look when fabricated. F5 Manufacturing Engineers oversee setup and implementation of these test subsystems from first PCA (Printed Circuit Assembly), and they are constantly refined. This allows us to compare and verify the final product to ensure it is what we expected it to be.

Beyond these safeguards, to better protect against the threat of unauthorized firmware and software manipulation, we embed two primary capabilities in our hardware: tamper detection and tamper prevention. The first is made possible by the inclusion of a TPM (Trusted Platform Module). TPM—ISO/IEC 11889—is a dedicated microcontroller that enables the verification of system integrity using cryptography. The second capability involves cryptographically signing subsequent firmware updates and validating them before installation, just as can be done with F5 software updates.

I would also like to call out the fact that F5’s IT organization owns and controls access to all hardware and networking of our test subsystems.

Questions for Your Vendor

When it comes to critical components of your network and security, trust and open communication should always be in the equation. While there is certainly ongoing debate around the facts in the Super Micro report, the type of threat it points to is very real and very credible. Regarding the potential for hardware and firmware attacks from any source—supply chain or otherwise—here are a few good questions to consider asking your vendor:

1. Do you use third-party developed motherboards in the design of your systems?
2. What, if any, part of the hardware design and development process do you control?
3. What level of security testing do you do on the hardware, and when?
4. What safeguards do you use to secure firmware and software updates?
5. Finally, what protections do you have in place to prevent and detect tampering from any source throughout the supply chain?

____

For additional information on how F5 can enhance your organization’s application security, please visit: https://www.f5.com/solutions/application-security