A note from Andy Ellis
Every security team has its horror stories of bad things that’ve happened to them. If you’re fortunate, you can learn from other teams by listening to these stories. If you’re really clever, you can use those stories to run your own tabletop exercises, learn from them, and hopefully teach your team some lessons along the way.
YL Ventures is curating incidents for you to use as tabletop exercises. Some of these stories are true. Some have the rough edges filed off of them. But all of them are useful as the seeds of tabletop exercises for you. Each exercise has been posted on one of our portfolio company’s blogs, and they’ve responded with how their solution might help you in that scenario.
Ready for the latest one? Let’s go!
Scenario:
It’s a normal day when you get the dreaded notification: You’ve been compromised. A DevOps engineer noticed slow performance on one of your internet-facing web servers, and upon inspection, found that it was running a cryptominer. You breathe a sigh of relief—after all, an adversary could have done much worse with that access (or is it a trap, and they left behind a cryptominer to distract you?).
Now forensics needs to start: how did they compromise the machine? It’s in the tier of systems that are the highest priority for patching, and there shouldn’t be any outstanding vulnerabilities. Is this a zero-day exploit? Who would burn a zero-day on installing cryptominers? That would be madness!
After deeper inspection, you find that a not-so-recent vulnerability—one you thought was patched—is present on the machine. Checking with your vulnerability management team, they show you the ticket that clearly indicates this machine was patched. What went wrong?
Digging deeper into your CI/CD system, you find the culprit. While the machine was patched, a later change reintroduced the vulnerable library. Now you have to look for other places where vulnerabilities have been reintroduced after you thought you’d patched them.
A New Hope: Opus to the Rescue!
Oh no! Nir Dagan here, DevSecOps Engineer at Opus Security. Let’s see what we can do to remedy this issue, help prevent similar attacks and improve overall security operations–particularly the patching process–so that a similar incident doesn’t occur in the future.
Assessing the Risk: The Phantom Unpatching
First, it's critical to know that the incident is isolated and unlikely to recur across other resources within the organization. Once the team identifies that the cryptominer exploited a specific vulnerability, it becomes imperative to assess the presence of this vulnerability across other services that may pose similar risks.
Opus leverages its comprehensive organizational mapping and Business Intelligence (BI) metrics to assign risk scores to different resources and services. By utilizing Opus's vulnerability view and applying targeted filtering options, security teams can easily identify all instances of the vulnerability within the organization. These results are meticulously sorted based on the severity of risk posed, ensuring a prioritized approach to remediation efforts. This systematic approach instills confidence that vulnerabilities are promptly addressed and patched organization-wide. Yee-haw!
Learning and Improving: Return of The Patch
The recurrence of the patched vulnerability was partly due to its discovery during operations on a live production workload, a phase that occurs late in the development and deployment pipeline. This discovery highlights a discrepancy with the shift-left approach the security team aims to implement, where identifying and addressing security issues earlier in the pipeline is crucial.
Recognizing the need to mitigate such risks proactively, there's a strong inclination to integrate a security scanner earlier in the development process. Specifically targeting infrastructure as code security issues, this scanner would conduct security checks during the Continuous Integration (CI) pipeline, flagging potential vulnerabilities before deployment. However, the decision to add additional security tools isn't without challenges; it entails both financial costs and operational overhead as teams must monitor and manage yet another tool and its outputs.
Implementing a comprehensive security platform often introduces complexities. Fortunately, Opus simplifies this process by seamlessly integrating with a wide array of security tools. Opus can ingest data from various sources, standardize outputs, and enrich information, presenting stakeholders with a unified interface. This capability significantly reduces the operational complexity associated with managing diverse security tools, fostering streamlined collaboration and decision-making across teams.
Building a Better Security Operation: The Force Awakens
The incident described above serves as a poignant example of a broader vulnerability in the company’s security operations. Historically, the decision to maintain separate application and infrastructure security teams was logical, given their distinct knowledge requirements and stakeholder interactions. However, in modern cloud-native environments, the interconnectivity between applications and infrastructure introduces new risks. A security gap in one area, such as an application vulnerability, can potentially serve as an entry point for attackers to exploit and compromise broader cloud infrastructure.
Recognizing the mismatch between operational reality and organizational structure prompts a reconsideration of approach. Opus offers a solution with a unified approach that accommodates the flexibility needed for various organizational types. By consolidating all security issues and vulnerabilities into a central platform, Opus eliminates silos that often hinder collaboration between different security teams or IT environments. This holistic approach streamlines operations by preventing duplication of efforts and saving valuable engineering time.
Opus empowers security teams to pinpoint the origin of vulnerabilities and address them at their source. This capability supports a shift-left strategy where security measures are integrated earlier into the development pipeline. By identifying vulnerabilities before they propagate into production environments, Opus ensures a more robust defense against incidents like the one described, where a supposedly patched vulnerability re-emerged post-deployment.
Our goal is to facilitate a proactive security posture that minimizes the likelihood of vulnerabilities resurfacing after updates or releases. This integrated approach not only strengthens defenses against cyber threats but also enhances operational efficiency and agility in responding to evolving security challenges.
Preventing Future Attacks: The Rise of Guardrails
A significant number of cyber attacks exploit known vulnerabilities, as evidenced by the incident involving this cryptominer. Beyond the initial incident response, preventing recurrence is paramount. Opus is designed to offer comprehensive visibility across diverse environments, equipped with robust vulnerability management and automation tools. This ensures proactive remediation of vulnerabilities based on their risk profile, enhancing efficiency and effectiveness.
Opus empowers security teams to navigate the increasingly complex landscape of vulnerabilities and cyber threats with confidence. By prioritizing and addressing vulnerabilities swiftly and strategically, Opus supports proactive defense measures that mitigate risks before they can be exploited.
