Uploaded March 2025 | Updated September 2026, 3 weeks ago
owasp2024globalappsecsanfra.sched.com/event/1g3an/hackuracy-boosting-ast-accuracy-through-hacking
How may a system's exposure to successful cyberattacks be detected more accurately? The short answer is that it is not possible with vulnerability scanning alone; expert manual evaluation by ethical hackers is also necessary.
While automated tools, due to their processing capacity and speed, have become indispensable in identifying potential vulnerabilities, they report high rates of false positives and false negatives, finding only 45% of systems' risk exposure. Tools cannot find vulnerabilities when their discovery involves an external user having come up with a complex, unexpected use of the application. This is where the expertise of ethical hackers comes into play.
In this talk, we will explain what accuracy in AppSec entails and specify three different measures that we used to assess security testing accuracy of scanning alone and the combination of scanning and hacking. We will characterize the insecure-by-design web application that was used as the target of evaluation (ToE), and then compare the performance of the different conditions both in reporting vulnerabilities and risk exposure (identified with a metric designed to accurately show the severity of vulnerabilities to help prioritize them for remediation).
We will present our research findings, which highlight that the combination of scanning and hacking dramatically outperformed all of the assessed tools' scanning in all three accuracy measures in identifying both the amount of vulnerabilities and associated risk exposure. Specifically, the combined approach achieved accuracy scores ranging from 78.9% to 93.7% in detecting the amount of vulnerabilities in the ToE, and from 94.3% to 98.5% in identifying its risk exposure. In contrast, the most accurate tool's scores ranged from 26.4% to 58.4% and 8.5% to 27.0%, respectively. Notably, the overall performance of application security testing in our research was better for guaranteeing few false negatives in reports than for providing reports containing all legitimate vulnerabilities.
In summary, we will demonstrate that achieving accurate detection of a system's risk exposure related to its vulnerabilities requires more than just automated security testing. It necessitates the involvement of expert hackers who can perform manual evaluations, understand the nuances of application logic and identify sophisticated security flaws. Further, we mention how the accuracy of vulnerability scanners can be enhanced. Ultimately, the goal is to equip developers, security professionals and organizations with the knowledge and tools needed to enhance the security of their applications and protect against threats.
-
Managed by the OWASP® Foundation
owasp.org
owasp2024globalappsecsanfra.sched.com/event/1g3an/hackuracy-boosting-ast-accuracy-through-hacking
How may a system's exposure to successful cyberattacks be detected more accurately? The short answer is that it is not possible with vulnerability scanning alone; expert manual evaluation by ethical hackers is also necessary.
While automated tools, due to their processing capacity and speed, have become indispensable in identifying potential vulnerabilities, they report high rates of false positives and false negatives, finding only 45% of systems' risk exposure. Tools cannot find vulnerabilities when their discovery involves an external user having come up with a complex, unexpected use of the application. This is where the expertise of ethical hackers comes into play.
In this talk, we will explain what accuracy in AppSec entails and specify three different measures that we used to assess security testing accuracy of scanning alone and the combination of scanning and hacking. We will characterize the insecure-by-design web application that was used as the target of evaluation (ToE), and then compare the performance of the different conditions both in reporting vulnerabilities and risk exposure (identified with a metric designed to accurately show the severity of vulnerabilities to help prioritize them for remediation).
We will present our research findings, which highlight that the combination of scanning and hacking dramatically outperformed all of the assessed tools' scanning in all three accuracy measures in identifying both the amount of vulnerabilities and associated risk exposure. Specifically, the combined approach achieved accuracy scores ranging from 78.9% to 93.7% in detecting the amount of vulnerabilities in the ToE, and from 94.3% to 98.5% in identifying its risk exposure. In contrast, the most accurate tool's scores ranged from 26.4% to 58.4% and 8.5% to 27.0%, respectively. Notably, the overall performance of application security testing in our research was better for guaranteeing few false negatives in reports than for providing reports containing all legitimate vulnerabilities.
In summary, we will demonstrate that achieving accurate detection of a system's risk exposure related to its vulnerabilities requires more than just automated security testing. It necessitates the involvement of expert hackers who can perform manual evaluations, understand the nuances of application logic and identify sophisticated security flaws. Further, we mention how the accuracy of vulnerability scanners can be enhanced. Ultimately, the goal is to equip developers, security professionals and organizations with the knowledge and tools needed to enhance the security of their applications and protect against threats.
-
Managed by the OWASP® Foundation
owasp.org
