Uploaded August 2026 | Updated September 2026, 2 weeks ago
There's already an increase in volume of security flaws found by LLMs. And orgs are already turning to LLMs to write code. So, what happens when orgs lean on LLMs to create patches for those security flaws? Keith Hoodlet gives an exclusive early look at his team's recent research into the success, quality, and failures of LLM-generated security patches. Notably, they saw scenarios across a spectrum from robust, effective patches to patches that changed the software's behavior to patches that introduced new vulns to patches that didn't even fix the original vuln while also introducing a new vuln.
The research considers factors like quality and correctness of prompts, complexity of the target software, programming language, and expertise required to understand what a robust patch should look like. If you're going to spend tokens on fixing security flaws, you want a feedback loop that fixes them correctly -- not an infinite loop of new flaws creeping in with every LLM iteration.
Watch for this research, its toolset, and data to be released on Thursday August 6th during Black Hat.
Visit securityweekly.com/asw for all the latest episodes!
Show Notes: securityweekly.com/asw-394
00:00:00 Episode Introduction, Announcements, and Guest Preview
00:02:45 Introducing the FLAWED Project on LLM Patch Quality
00:05:37 Understanding the Five Outcomes of LLM-Generated Patches
00:07:46 The Challenge of Achieving Robust LLM Patches
00:11:56 Exploring Modes and Prompts in LLM Patching Research
00:17:12 How LLM Attention Mechanisms Lead to Fragile Fixes
00:21:03 Analyzing LLM Patching in Real-World Scenarios
00:26:22 The Role of Formal Verification in Future Patching
00:31:55 The Financial and Human Cost of Validating LLM Patches
00:36:53 Balancing LLMs for Code Understanding and Patching
00:39:01 Next Steps for Improving LLM-Generated Patch Outcomes
00:44:16 Analyzing the Hugging Face LLM Security Breach
00:54:17 The Inertia of Updating to Secure Fast Jason Versions
01:00:09 A Manager's Approach to Empowering Developer Teams
There's already an increase in volume of security flaws found by LLMs. And orgs are already turning to LLMs to write code. So, what happens when orgs lean on LLMs to create patches for those security flaws? Keith Hoodlet gives an exclusive early look at his team's recent research into the success, quality, and failures of LLM-generated security patches. Notably, they saw scenarios across a spectrum from robust, effective patches to patches that changed the software's behavior to patches that introduced new vulns to patches that didn't even fix the original vuln while also introducing a new vuln.
The research considers factors like quality and correctness of prompts, complexity of the target software, programming language, and expertise required to understand what a robust patch should look like. If you're going to spend tokens on fixing security flaws, you want a feedback loop that fixes them correctly -- not an infinite loop of new flaws creeping in with every LLM iteration.
Watch for this research, its toolset, and data to be released on Thursday August 6th during Black Hat.
Visit securityweekly.com/asw for all the latest episodes!
Show Notes: securityweekly.com/asw-394
00:00:00 Episode Introduction, Announcements, and Guest Preview
00:02:45 Introducing the FLAWED Project on LLM Patch Quality
00:05:37 Understanding the Five Outcomes of LLM-Generated Patches
00:07:46 The Challenge of Achieving Robust LLM Patches
00:11:56 Exploring Modes and Prompts in LLM Patching Research
00:17:12 How LLM Attention Mechanisms Lead to Fragile Fixes
00:21:03 Analyzing LLM Patching in Real-World Scenarios
00:26:22 The Role of Formal Verification in Future Patching
00:31:55 The Financial and Human Cost of Validating LLM Patches
00:36:53 Balancing LLMs for Code Understanding and Patching
00:39:01 Next Steps for Improving LLM-Generated Patch Outcomes
00:44:16 Analyzing the Hugging Face LLM Security Breach
00:54:17 The Inertia of Updating to Secure Fast Jason Versions
01:00:09 A Manager's Approach to Empowering Developer Teams
