Detecting the SolarWinds Hack –
Stel Valavanis
The cybersecurity world has been overtaken with concern over a state-sponsored cyber attack, perpetrated by Russian intelligence agents, against multiple federal agencies including those responsible for our nuclear stockpile, and prominent cybersecurity firms such as Microsoft and FireEye, who were the first to identify the attack.
Their internal networks were accessed, undetected, since March of 2020, and the attackers accessed documents, stole penetration testing tools, and found their way into other systems. The attackers initially inserted a maliciously modified version of SolarWinds Orion, a tool used by many organizations, both private and public, to monitor large networks, into the SolarWinds update server as early as March of 2020. As the recent investigation began, it was clear that this trojanized update, called SUNBURST, had spread widely, though it still is difficult to know if the back door the hackers created for this attack allowed further intrusions and infection. It is known that up to 18,000 customers of SolarWinds have been affected by the malware. More will certainly be found out in the coming weeks. The targets being high profile, high in number, and the novelty of the supply chain attack vector are enough to fill the papers, but customers, clients, and everyone in the ecosystem are left with many questions on what the lasting effects of this hack will be. It is being called the most significant cyber attack in our history.
Beyond learning quite a bit about how the attack on our federal agencies and companies via the SUNBURST malware was actually perpetrated, businesses and cybersecurity professionals are asking what lessons can be taken away. How will this change how we do things? What things should organizations and cybersecurity practitioners consider? Many companies, both those affected by the hack and those not (and it’s not 100% clear who is who at the moment) have released statements with some recommendations, such as onShore Security did for its clients this week:
SolarWinds SUNBURST Incident Response
On December 8th, 2020 it was announced that security vendor FireEye had been compromised by malicious actors, and several specialized software tools for performing network testing and security penetration had been taken. On December 13th and 14th 2020, FireEye and software vendor SolarWinds issued a statement that the SolarWinds Orion monitoring software had been compromised by state sponsored malicious actors by inserting a malware backdoor (known as SUNBURST) into a critical .dll file. The insertion of this malicious code into SolarWinds Orion is believed to have occurred between March and May of 2020. SUNBURST allows malicious actors remote control of the SolarWinds Orion software, as well as the ability to utilize the software as a vector for critical data exfiltration. Additionally, a host running the compromised software potentially becomes a jump host for lateral exploitation of other resources within an enterprise perimeter.
onShore Security began updating its Panoptic Sensors and Panoptic SIEM with detection capabilities focused on the FireEye tools which had been maliciously taken on December 11th, 2020. On Sunday, December 13th, onShore Security began updating its Panoptic Sensors and Panoptic SIEM to detect SUNBURST indicators of compromise, and have updated its threat intelligence lists to blackhole known traffic associated with this compromise. At this time, all onShore Security Panoptic Cyberdefense clients have detection in place for the SUNBURST malware.
On Monday, December 14th 2020, the US Cybersecurity and Infrastructure Security Agency (CISA) issued an advisory that all US federal civilian organizations to disconnect and power down their SolarWinds Orion installations. At this time, onShore Security concurs with this recommendation, and urges all private and public organizations to follow suit. Given the highly sophisticated nature of this attack, and its use of stenography to mask data exfiltration, onShore Security feels it is too soon to certify any SolarWinds code as clean, or unaffected. Additionally, as SolarWinds operates primarily as an infrastructure monitoring and management platform, customers with this software installed should consider all credentials (domain admin, router/switch, SNMP community strings, etc) utilized by this software as compromised. Therefore, all critical administrative credentials should be changed as soon as possible. Finally, onShore Security recommends that all of its clients follow best practices for vendor risk management, and contact any critical vendors (particularly those which may have access to PII, PHI, or business critical data) which may be utilizing SolarWinds Orion.
At this time, onShore Security cannot fully assess the risk of this compromise, as this type of software backdoor may have allowed malicious actors to place additional malware within client networks which could dwell for several months prior to activation. Overall, we recommend clients work to limit outbound network traffic to only necessary global regions or locations, as well as continue to monitor for abnormal traffic and file access. As a Managed Detection and Response provider, onShore Security is committed to continuing monitoring for abnormal or malicious activity within its clients’ resources. onShore Security does not utilize SolarWinds software anywhere in its network or systems.
Steven Kent
Chief Technology Officer
onShore Security
In the very short term, care should be taken regarding these exact pieces of software. If SolarWinds software is being used in your organization, it’s recommended that it be shut down and not turned back on until SolarWinds publishes a 3rd party code audit that makes it clear that the vulnerability gaps are filled. Secondly, your organization should adopt the rules released to detect for SolarWinds’ vulnerability and use the signatures provided by FireEye to detect SUNBURST and the FireEye tools stolen in the attack.
The nature of the attack, that of writing a backdoor into the software, and the unusually long dwell time, mean that it is simply impossible at this point to be able to clear anyone of risk and the possibility of other, secondary “infections” must be considered until dispelled.
Because of the long dwell time, lateral movement in the network is almost assured. This teaches us an important lesson in protection vs. detection. When protection and prevention fail, it can be impossible to know until it is far too late. In the case of this hack, the attackers had months inside networks, allowing lateral movement, secondary infection, and other malicious activity that may require deep forensic investigation to uncover and repair. Many of the organizations that have been affected are still trying to make up for lost time in their efforts to mitigate and prevent similar attacks. For this reason, MDR services such as our own Panoptic Cyberdefense are critical. They not only reduce dwell time, they also provide valuable forensic data that serves as a focal point for mitigation efforts and future plans.
We must also consider the software supply chain. Reliance on 3rd party suppliers of software is only increasing and our government has been even more willing to trust 3rd parties, as it moves capability to the cloud and other forms of technology outsourcing. Some attention has been paid to hardware supplied by Chinese companies, but this event clearly shows that attackers don’t need to own pieces of the supply chain to infect it. Scrutiny needs to be applied where it can be and, in the case of software, that means code review. Using open source options where possible makes review easier because a wide array of parties can collaborate on the review. APIs need to be published and open. Our government can contract from suppliers but require that all components and licenses meet the required certifications. Famously, the Chinese government insisted that Microsoft provide source code to them for review as a prerequisite to doing business in China. That’s a tall order, but their fears weren’t unfounded. This year, it was revealed that a Swiss company supplying secure communications to many governments included a back door for our own CIA, demonstrating a need for some sort of cyber-arms treaty.
We also need to see greater collaboration with the detection process. SolarWinds had been instructing clients to exclude certain Orion binaries from anti-malware scanning because false positives were produced. This is likely at least one reason the attackers chose those binaries. There are reasons for exclusions but often it’s a way to avoid the harder task of collaborating with anti-malware and detection vendors to supply appropriate signatures for proper scanning. Microsoft, who were one of the victims, quickly revoked the digital certificate for the malicious binaries but clearly more care must be put into the signing and verification process as well.
The unfortunate truth is that there exists a zone of uncertainty around this hack. It is easy to tell if you were targeted in any way, meaning you can tell if the attackers ever took notice of you. Beyond that, the extent of the attack on you in particular can be hard to suss out. There are some signature parts of the attack that can be searched for. For example, email systems were frequent targets. Also, instances of create, execute, delete commands can be evidence of the malware covering its own footsteps. These novel stealth tactics, designed to avoid detection and increase dwell time as much as possible, mean that forensic investigators have their work cut out for them, now put into the position of, essentially, proving a negative.
In the future, supply chain attacks will be part of any organization’s threat modeling and there will be policy in place to detect and even prevent similar attacks. If anything, however, this incident highlights the importance of detection in the cybersecurity stack, the need for greater scrutiny in the code or signing of software, using open source and open APIs where possible, and the need to begin serious work on cyber diplomacy.