Vulnerability Management - A Story

Nir Dagan, DevSecOps Engineer

August 3, 2024

5 min read

New Company Jitters

It is always exciting to join a new company. No matter how experienced you are, there are always those jitters of new beginnings. However the transition was not limited to our team, the company itself was changing from a small startup to a mid-size company, and as it grew the customer base was changing as larger and more established businesses were starting to use the product.

It was exciting to be in the company during that spurt of growth. There was a sense of optimism and confidence. On our part, we felt that leadership was ready to put more focus and resources on product and infrastructure security.

The pace was staggering; every week new features were added to the product, and new cloud accounts were opening on a daily basis.

My team’s responsibility was to secure the different services and products the company offers to its customers. Considering those services were all cloud-native applications, that meant that we needed to have security controls and processes in each stage of the development and deployment cycle. Considering the size of our company, we did not have an AppSec and infrastructure team. This meant in addition to supporting the different SDLC processes, the team would be responsible for security of the cloud infrastructure that was used to power the company services.

Know the Unknown

Security is a multi-layered process. A company can check all the boxes and still not have a mature security program. Attributes like secure architecture, culture, and safeguards are often not captured well in compliance checkboxes and security products. However, there is immense value in those processes as they create a road map and help organize resource allocation for different security projects. One of the main initiatives is the resolution and management of vulnerabilities in different IT systems and software services. According to NIST, a vulnerability “is a weakness in an information system, system security procedures, internal controls, or implementation that could be exploited or triggered by a threat source.”

When discussing software security, one of the leading sources is the OWASP project. We started looking for some papers and recommendations on software security processes. Here is a diagram showing the process:

As you can see, there are three main cycles : Detection , remediation, and reporting. From https://owasp.org/www-project-vulnerability-management-guide/

Detection

First, we had to ensure we were able to detect the vulnerabilities. We used a variety of tools, some commercial and some open source. It was important for us to identify vulnerabilities across the different stages of the development pipeline and for services that were already in production.

Using the GitHub API we made sure whenever a new repository was created in our organization, we would get a notice. We then created a small questionnaire about what the repository was used for, and later used this information when we needed to prioritize the vulnerability.

Every repo containing code scheduled for integration into our product must undergo a vulnerability scan as part of the Continuous Integration and Continuous Deployment (CI/CD) process

Since all our services would only be deployed on major cloud providers platforms, we had to ensure that all our accounts had a vulnerability scanner deployed. The amount of vulnerabilities discovered were in the thousands. We understood that not all were of equal importance, and we knew that we needed to prioritize.

Prioritizing

In order to establish a structured approach for addressing these vulnerabilities, we decided to categorize and group them and communicate this process to other stakeholders beyond our team.

1. Categorizing

We categorized each vulnerability into five different types and designed an SLA for each, including an agreed time for resolution. For example, a supercritical vulnerability like log4j would require us to stop everything we were doing until it was fixed, while other vulnerabilities would be categorized as less urgent.
Upon completing the process, we developed a quadrant system based on two key factors: the simplicity of the fix and the criticality or severity of the vulnerabilities. We then collaborated with the relevant stakeholders to establish Service Level Agreements (SLAs) for addressing each category of vulnerabilities.

2. Determining the Criticality

We started with the CVSS score from the scanner as a baseline, and added our own environment context to create a more representative score for our business. If the vulnerability severity score was very high, we would do some of our own technical analysis of the vulnerability to see if it was exploitable in our current setup. However, running an analysis like that on each vulnerability was not scalable, so we created a list of questions/attributes that would help us understand the score.
These are a few examples:

How many hops are needed to reach the system running the vulnerability (public 0 hops , company domain 1 hop and extra)?
What is the type of environment? Is it dev, production, or a lab?
What is the system risk to the business - loss of data, disruption of business, disruption of corporate activity?
Can the system be used as an hop to another system - Jenkins server -> cloud account, Workstation -> database , etc.?
Can the vulnerability affect the company customer - weakness in authentication, loss of customer data?
How likely is it to use the exploit in our current environment?

Once we added our input, we wanted to create a decision tree similar to an SSVC score, but we wanted to add another parameter that would influence our final decision - ease of fix. This parameter, in broad terms, helped us understand the complexity involved in rectifying the vulnerability.

We used some of the following attributes to create a score for remediation complexity:

What are the consequences of the systems/app being unavailable?
How many versions of upgrades would be needed to reach the fix candidate?
Does the fixed candidate upgrade require minor or major upgrades?
Does the system have a vendor that supports it or is it an obscure open source project?
How well maintained is the source code?
How often, in how many systems, does the vulnerability appear? This can actually go both ways, making it more worthwhile to address and also more complex because it can affect many systems.
How many times has the library been called in the code? This requires technical understanding of the code.

It wasn't perfect, and we knew sometimes assumptions would be wrong. However, we felt that this process gave us a few advantages. By using junior members of the team to process each finding, it brought overall cost down, and allowed more senior members to focus on strategy. Instead of having a number that could be interpreted in different ways, we had an action with an accepted SLA. It gave us a baseline to set the expectation for time-till-remediation.
We could use the information added by the process to create detailed reports. We could test and change our assumptions as we got more feedback from the product system owners.

3. Creating Tickets for Code Owners

We were pretty happy with the process we built. The next phase was to process the output of our different scanners and create tickets for code or system owners. This part of the process was passed from our team to engineering. The expected behavior was known only to the code or system owners, requiring them to conduct the code tests.

We started creating tickets and eagerly anticipated the gradual resolution of vulnerabilities within our code, and we were expecting to be able to proceed to other important tasks. We did have some success with our infrastructure teams; however, it soon became clear the developers did not share the same sense of urgency.
Our ticket responses ranged from pushing out due dates to outright disregard, yet the one response we never received was, "issue resolved." It was obvious that we were not moving forward, definitely not at the pace we wanted. One option was to accept the reality that this organization didn't prioritize security, and occasionally, only when a sufficiently critical vulnerability came up, some team leader might address our concerns. However, we recognized that this approach was not sustainable, and we needed a change in the processes to drive change.

Negotiate, Enforce, Renegotiate, and Repeat

At our company, the work was divided into two-week sprints. Every feature would be designated into tasks, each with a maximum completion time frame of two weeks.

At the end of each sprint, the developer team leads and the product leads would meet to discuss what work would be done in the next sprint. We knew throwing tickets into the wind wasn’t going to help resolve vulnerabilities. We knew we had to join these meetings, and we did.

The product team plays a key role in remediation. They assign the tasks for each development cycle. Our teams agreed that each cycle, or every other cycle, will have time allocated to deal with fixing issues in the code. This meant that the security team had to act as a customer and push the product team to allocate time.

In addition to assigning tasks, the product team also categorized engineering work into attributes. Each sprint incorporated different attributes.Things like: reliability, extensibility, and performance. It is a way for the company to manage its technical debt, especially when there is a race toward creating new features. Security wasn’t taken into account, until we created a more collaborative work environment. After security was a category, the company now had a way to measure risk posture.

The prioritization process we created earlier came in handy, because we had a way to articulate what vulnerabilities needed to be fixed and why. During each meeting, we presented a list of vulnerabilities and security issues that were most critical. We also had a general understanding of the complexity and could occasionally point to other teams that performed the fix. This allowed us to manually create an organizational database of patching experiences.

Reporting

The third part of the OWASP diagram is the reporting cycle. One of the keys for success in this process is having a consistent way to track and report your progress. We had to make sure all of the key stakeholders were aware of the progress or lack of it. Using the reports we could track if the product was getting safer as time went by or if vulnerabilities were accumulating faster than we were repairing them. We could see which teams were efficient in fixing vulnerabilities and which weren't.

The reports were the only way we could make our case for additional resources or in cases where we were making progress to demonstrate our success. We discovered that the number of vulnerabilities and difficulty in addressing them correlated strongly with the general lack of maintainability of the software. So it happened that our report was an indicator that the CTO used when deciding which projects needed upgrades or redesign.

Thankfully, Things are Different Now

As I recount the process my team and I built at my previous company, I’m both proud and relieved. What we built was painfully manual. Every step along the way - from prioritization to reporting - was done manually and kept on sophisticated spreadsheets. It was inefficient and prone to human error.

Today’s world of vulnerability management and remediation needs something drastically different. That’s the reason I moved to Opus Security. Our cloud-native, remediation platform aggregates, deduplicates, and prioritizes vulnerabilities across AppSec and Cloud Sec, reducing the complexity of remediation into a single unified view. If I had Opus Security in my previous role, I could have decreased the number of human hours from half a day to hours, and in some cases, minutes.

See the Opus Platform in Action

Cybersecurity Consolidation: The Key to Stronger Security

February 13, 2025

SSVC Explained: How It Transforms Vulnerability Prioritization

February 5, 2025

How to Achieve and Maintain FedRAMP Compliance

February 2, 2025

Cloud-to-Code Integration: Enhancing Security from Cloud to Codebase

January 25, 2025

Cloud Security Posture Management (CSPM): A Comprehensive Guide

January 14, 2025

Remediation as Security Immunization: Protecting Against Exploits, Malware, and Ransomware

January 25, 2025

Security Remediation Insights - Looking Back at 2024, Ahead to 2025

January 15, 2025

No-Code Automation in 2025: Choosing and Implementing Tools

January 14, 2025

Build vs. Buy: A Guide for Security Remediation Platforms

January 14, 2025

How To Design an Effective Vulnerability Management Program

January 14, 2025

Exposure Assessment Platforms (EAPs): A Cybersecurity Essential

November 20, 2024

NYDFS Cybersecurity Regulation: Key Compliance Steps

January 14, 2025

Is Python the Right Tool for Your Cyber Security Strategy?

January 14, 2025

What Is DORA Compliance? A Practical Guide

January 14, 2025

Opus' Ownership Detection & No-Code Workflow Builder

November 5, 2024

Remediation vs. Mitigation: Key Differences Explained

November 7, 2024

What Is Vulnerability Remediation & How It Works

January 14, 2025

Vulnerability Assessment: Definitions, Methodology & Tools

November 7, 2024

Wrapping Up Season One of "30 Min on: Vulnerability Management"

October 21, 2024

CVE and CVSS: Vulnerability Assessment & Severity Ratings

October 29, 2024

CVSS Scores: How They Enhance Vulnerability Assessment

February 12, 2025

What CVE Scores Mean for Your Cybersecurity Strategy

October 29, 2024

The 2025 Guide to Risk-Based Vulnerability Management

January 12, 2025

Secure SDLC: Why and How to Integrate Security In Your Development Life Cycle

October 20, 2024

Application Security Posture Management (ASPM): How It Works & Top Tools in 2025

January 12, 2025

How to Prioritize Vulnerabilities Effectively in 2025

January 12, 2025

The Opus Multi-Layer Prioritization Engine: What is it All About?

September 10, 2024

EPSS Score: Improving Vulnerability Prioritization in 2025

January 12, 2025

Opus Platform Explained: Cloud-Native Vulnerability Remediation

October 20, 2024

It’s Time to Get Proactive About Vulnerability Remediation

October 29, 2024

The Vulnerability Strikes Back - TABLETOP PART TWO

August 13, 2024

Opus Security Joins CrowdStrike Marketplace

September 29, 2024

Cloud-to-Code Integration for Comprehensive Remediation

January 15, 2025

Understanding CTEM: How Opus Security Can Help

September 22, 2024

This Is What Your Security Champions Program Is Missing

September 29, 2024

Risk-Based Vulnerability Management (RBVM) in a Cloud-Native World

October 7, 2024

How Business Impact Analysis Facilitates Effective Remediation

October 20, 2024

[Case Study] How Opus Solved the Puzzle of Disparate Tools and Alerts

October 7, 2024

It’s Time for Security Operations to Ditch Excel

February 13, 2025

Security Remediation Budgeting - Everything You Need to Know

October 20, 2024

How to Achieve More Effective Vulnerability Management?

August 13, 2024

Checklist: Tracking Down Key Stakeholders for Effective Security Remediation

October 2, 2024

Streamlining Security Remediation: The Convergence of CTEM and ASPM into a Single Operating Platform

September 22, 2024

How to Locate Key Stakeholders for Security Remediation

October 7, 2024

The Evolution of Vulnerability Remediation: From RBVM to Security Remediation Platforms

September 29, 2024

Practical Vulnerability Remediation - When Should You Act?

October 20, 2024

The Ultimate Remediation Lifecycle for Vulnerability Management

September 22, 2024

Global Leader in Content Solutions Streamlines Remediation with Opus

August 13, 2024

Opus Security helps Global Technology Company Connect Security Tools for More Effective Security Remediaiton

August 13, 2024

Opus Security's VP of Marketing shares her thoughts on B2B marketing in a recent DemandBase podcast.

August 13, 2024

Opus Security Wins 2023 CyberSecurity Breakthrough Award

October 20, 2024

Vulnerability Management - A Story

September 22, 2024

6 Ways Orchestration Reduces Cloud Security Risk

October 2, 2024

Move Beyond Shift Left and Shift Smart

October 20, 2024

3 Tips for Bridging the Remediation Developer/Security Gap

October 2, 2024

WAF With EKS, Using Only k8s Controllers

August 13, 2024

Opus Recognized in the 2023 Gartner® Emerging Tech: Security — The Future of Cyber Is Automated Moving Target Defense report.

August 13, 2024

Wiz identifies, Opus fixes.

August 13, 2024

Writing your Cloud Opus: A Deep Dive into Orchestrating your Cloud Security

August 13, 2024

Snow in April, Alert Monsters, and Connect Four: 5 Takeaways from our SecOps Chat

August 13, 2024

Security in Sync: How Security Teams Can Make Cloud Remediation Less Painful for Engineers

October 20, 2024

Why We Need a New Approach to Remediation in the Cloud

August 13, 2024

Introducing Opus: Orchestrating Remediation and Reducing Risk in the Cloud

September 29, 2024

Vulnerability Management - A Story

Read next

See the Opus Platform in Action