How coding agents can transform the experience of using complex security analysis tools.

I’ve been exploring how coding agents can enhance existing software, and today I’m excited to share Patchsmith - a project that aims to transform the user experience of an amazing, albeit quite complex, piece of software: CodeQL.

CodeQL is GitHub’s semantic code analysis engine that powers their Advanced Security offering. It’s incredibly powerful for finding security vulnerabilities and code quality issues, but it comes with a steep learning curve and significant operational overhead. Patchsmith uses AI agents to make CodeQL more accessible and effective.

The challenges of Static Application Security Testing (SAST)

Before diving into the solution, let’s understand the problems that organizations face when adopting static code analysis tools like CodeQL:

1. Fine-tuning is Required

Generic scanning engines need to be customized for your specific domain and codebase. This means writing custom CodeQL queries to identify:

• Your application’s specific data sources (user inputs, API endpoints, database queries)
• Your sinks (where data is consumed or output)
• The flows between sources and sinks that matter in your context

Writing effective CodeQL queries requires deep knowledge of both the query language and your codebase architecture. It’s a non-trivial, time-consuming process that requires specialized expertise.

2. False Positives Plague Rule-Based Analysis

Even well-tuned static analysis tools generate false positives. Rule-based analysis operates at the local code level and often lacks the broader context needed to determine if a finding is genuinely exploitable.

Every alert requires careful manual review by engineers to:

• Understand the data flow
• Assess whether the vulnerability is reachable
• Evaluate the actual risk level
• Decide whether to fix, suppress, or defer

This triage process is time-consuming and prone to inconsistency across teams.

3. Fixing Takes Developer Time

For each legitimate issue identified, developers must:

• Understand the vulnerability and its implications
• Research appropriate remediation approaches
• Implement the fix
• Test the changes
• Prepare a pull request

When you multiply this effort across dozens or hundreds of findings, it becomes a significant burden on engineering teams, especially when dealing with accumulated technical debt.

How Coding Agents Address These Challenges

This is where AI agents shine. They can:

Generate custom queries by reading and understanding your codebase structure, identifying patterns of sources and sinks specific to your application.

Review security reports (SARIF files) to eliminate false positives by analyzing findings in their broader context, assessing exploitability, and prioritizing based on actual risk.

Write fixes automatically by understanding the vulnerability, researching remediation patterns, generating appropriate code changes, and preparing pull requests.

The research backs this up. The IRIS study demonstrated substantial improvement in CodeQL detection capabilities by using LLMs to write custom queries and review findings:

The results show that LLM-enhanced analysis significantly outperforms baseline CodeQL configurations across multiple vulnerability types.

Introducing Patchsmith

Patchsmith is a CLI tool that brings these AI agent capabilities to CodeQL. It’s designed to fit into your existing development workflow while dramatically reducing the expertise and effort required to get value from static analysis.

Installation

Getting started is simple:

pipx install "git+https://github.com/eschnou/patchsmith.git"

Core Capabilities

Initialize and Analyze

# Set up Patchsmith for your repository
psmith init

# Run CodeQL analysis
psmith analyze

These commands handle the CodeQL setup, database creation, and initial analysis - all the boilerplate that normally requires reading extensive documentation.

Fine-tune with Custom Queries

# Generate custom queries based on your codebase
psmith finetune

# Focus on specific security concerns
psmith finetune --focus "SQL injection in user authentication"

# Target specific languages
psmith finetune --language java

# Control the number of queries generated
psmith finetune --queries 5

The finetune command analyzes your codebase to understand its structure, identifies relevant sources and sinks, and generates custom CodeQL queries tailored to your application. The agent considers:

• Your application’s architecture and frameworks
• Common data flow patterns in your code
• Security-relevant interactions
• Language-specific vulnerabilities

Investigate Findings

# Get detailed analysis of a specific finding
psmith investigate F-3

Instead of just showing you the raw CodeQL output, Patchsmith’s agent:

• Explains the vulnerability in plain language
• Analyzes the data flow and context
• Assesses exploitability and risk
• Provides recommendations

Generate Fixes

# Automatically generate a fix for a finding
psmith fix F-3

The agent will:

• Understand the root cause
• Research appropriate remediation patterns
• Generate a code fix
• Prepare it as a pull request (coming soon)

Real-World Impact

Here’s what this looks like in practice:

Without Patchsmith:

1. Security team runs CodeQL scan (requires CodeQL expertise)
2. Results contain 150 findings
3. Engineering team manually reviews all 150 (takes days/weeks)
4. Team identifies 30 legitimate issues after triage
5. Developers spend weeks implementing fixes

With Patchsmith:

1. Run psmith analyze (no expertise required)
2. Run psmith finetune to add custom queries for your domain
3. Agent triages findings, identifying 30 legitimate issues with risk assessment
4. Run psmith fix to generate pull requests for straightforward fixes
5. Developers review and merge fixes, focusing on complex cases

The time savings are substantial, but more importantly, you get better results because the AI agent can consider context that rule-based analysis misses.

The Broader Vision

Patchsmith is part of a larger trend: using AI agents to make powerful but complex tools accessible to broader audiences. There are countless sophisticated CLI tools that deliver tremendous value but have steep learning curves:

• Security tools (like CodeQL)
• Infrastructure tools (Terraform, Kubernetes)
• Database tools (migrations, query optimization)
• Build and deployment tools

Each of these could benefit from “agentic integration” - wrapping them with AI agents that:

• Simplify configuration and setup
• Generate domain-specific customizations
• Interpret results in context
• Automate routine remediation

This is what I’m calling “Tool-Enhanced AI” - using AI not to replace tools, but to make them dramatically more effective and accessible.

Get Involved

Patchsmith is open source and I’d love your input:

• Try it out: github.com/eschnou/patchsmith
• Report issues: Share bugs or feature requests on GitHub
• Join the community: Connect on Discord or Bluesky
• Share use cases: I’m especially interested in real-world scenarios and improvement opportunities

Whether you’re dealing with security findings in your codebase or just curious about how AI agents can enhance existing tools, I’d love to hear from you.

Static analysis doesn’t have to be this hard. Let’s make it better together.