Cloudflare's Security Audit Skill on a Local Qwen: 9 Runs, 5 Real Findings, 4 False Alarms Rated High

On Nostr, Juraj recommended Cloudflare’s security-audit skill for every project you run, free models included. His point: the skill is model-independent, it works in any agent harness, and it finds more even in code that a model has already reviewed. I wanted to know what that looks like when the model is not a frontier model but the one I run at home, on a single machine, with no API bill.

So I pointed it at my own code: sovereign-fleet (the task runner for coding agents I use every day, compared with other orchestrators here), Vozonda (a self-hosted text-to-podcast app) and the repository behind this site. Before that, I tested whether the setup finds bugs I already knew about.

The setup

The skill is a set of instructions plus two small Node validators. It walks an agent through six phases: map the architecture and trust boundaries, pick the attack classes that apply, audit each one into a coverage ledger, then try to disprove its own findings before it reports them. Every finding ends up as confirmed, needs_validation or rejected, with likelihood and impact.

I used the skill at commit c1c8a8c (2026-09-14) and ran it with:

• Model: Qwen3.6-35B-A3B (NVFP4) on vLLM, on an NVIDIA DGX Spark. The same model my agent fleet uses for most of its work, and the one behind my harness bake-off.
• Harness: opencode 1.19.5, opencode run --auto.
• Sandbox: each run in a Landlock sandbox with write access only to a scratch directory, no network, no credentials. The target was a plain export of the repository, never the working copy.
• Mode: source only. The prompt told the agent not to build, run or install anything; anything that would need execution stays needs_validation.

Each prompt named the scope and a few focus areas for the project (for Vozonda: unauthenticated routes, the feed key, SSRF and DNS rebinding in link reading, data sent to third parties). That is a hint, and it matters for one result below.

Calibration: two known bugs, three blind runs

Earlier that week another coding agent had written a feature for Vozonda: read a source’s Lightning address, including from the author’s Nostr profile. Its first draft had two real security bugs that I had found in review:

1. Privacy leak. Every source link, not only Nostr ones, triggered a profile lookup, so the URL of every article a user turned into an episode went to public Nostr relays.
2. Trust. The lookup accepted any profile event a relay returned, without checking the event id, the signature or the author’s key. A relay could hand in its own Lightning address.

I gave Qwen that draft, scoped to the four files involved, three times, without mentioning either bug. That is six chances to find something I already knew.

One hit out of six. The privacy leak, which I consider the worse of the two, never came up. Run 1 is the uncomfortable one: an audit that reads the vulnerable code and calls it correct is worse than no audit if you trust it.

Run 3 is the reason I kept going. Vozonda checks every link it fetches against private and local addresses, then lets httpx open the connection. httpx resolves the name a second time. A host that answers with a public address for the check and with 127.0.0.1 for the connection gets past the guard. I had not seen that, and it was real.

Six audits on three repositories

After the calibration I ran two full audits per repository, one after the other on the same machine.

Of 21 findings the model marked confirmed, 4 survived my review. One run (sovgrid.org, run 1) never wrote its output files; its leads exist only in the session log. Total GPU time for all nine runs was a little under three hours, and the API cost was zero.

The real findings

The fixes are small. Vozonda now connects to the address it checked, with the original name in the Host header and in the TLS check. In sovereign-fleet 0.6.7 the MCP client trusts exactly the daemon’s own certificate when it talks to an IP, and its egress proxy refuses a name with any private answer and connects to the address it checked. The site’s /api/* routes got the same per-IP limit the MCP endpoint already had.

To be fair to my own code: the egress limit was written down in the module’s docstring as a known limitation. The model may simply have read it. It still counts, because the docstring was a note, not a fix.

The false alarms

The false alarms were not random. Most came from judging one route without reading what runs before it.

The worst: Vozonda, run 2, rated critical that GET /distribution hands out private feed URLs with their access key to anyone. It does return those URLs, to the owner. Every request from another device passes a middleware first that rejects it without the token. The model never connected the route to that middleware. Run 1 reported the same route and the settings route as two separate high findings.

Other patterns:

• Design decisions as vulnerabilities. “The owner can point the app at any Nostr relay” was rated an SSRF. It is a setting, behind the login, and pointing it at your own relay is the purpose.
• Missing context. An “open redirect” on the route that links an episode to its source: private episodes are gated, public ones redirect to the URL the owner entered.
• Headers for the wrong content. A missing Content-Security-Policy on an endpoint that only returns JSON.
• Operator input as attacker input. A file path read from an environment variable that only the operator sets.

Across the six audits, four findings rated high or critical turned out to be false. Three of the real ones were rated high; I rate them low to medium. If you sort the report by severity and fix from the top, you spend your first hour on things that are fine.

What the hint did not do

Vozonda’s audit prompt named DNS rebinding as a focus area. Neither of the two Vozonda runs reported it. The calibration run, whose prompt did not mention it, found it. With three data points that proves nothing about hints, but it is a clear reminder that one run is a sample, not a verdict. Two runs on the same code disagreed on almost everything except the sovereign-fleet pair, which found the same two real issues.

So, is Juraj right?

Partly, and the part that holds is the important one. The skill turned a 35B model on my desk into something that found five real issues in code I had already reviewed, for the price of electricity. The structure does a lot of the work: the coverage ledger forces the model through attack classes it would otherwise skip, and the validators kept every run’s output well-formed. Nothing in the setup needed a cloud model.

What does not hold with a model of this size is the verification phase. The skill asks the agent to try to disprove its own findings, and a smaller model is not good at that. It confirmed 17 findings that a few minutes of reading disproved, and once it confirmed vulnerable code as correct.

How I will use it from now on:

• Run it several times per scope, not once. The runs disagree, and the union is more useful than any single run.
• Treat confirmed from a small model as a lead. Every finding gets a human check, or a check by a stronger model, against the actual code path.
• Calibrate first. Plant or reuse two known bugs and see how many come back. One out of six told me exactly how much weight to give the rest.
• Read the noise for patterns. Three false alarms rated high or critical rest on the same misunderstanding (a route judged without its middleware). That points at context the model did not load, not at three separate problems.

A suggestion for the skill itself: a step that makes the agent list the global request pipeline (middleware, auth dependencies, proxy rules) before it rates any single route would have removed most of my false positives.

Run it on my code

The repositories in this test are public: sovereign-fleet and Vozonda. If you run the skill on them, with any model, I want to see what you find. Please report through GitHub’s private vulnerability reporting (the Security tab of each repository), not in a public issue, and please audit the code, not the servers that run this site.