Stephan Schmidt - November 7, 2026
No Code Review. Fuzz Instead.
Nobody can read all the code agents write, and now marketing wants to ship too. Copy the app store, put every app into a WASM sandbox and fuzz every deploy.
TL;DR: Agents write more code than anyone can review, and an AI code review on every pull request gives different answers each time and follows instructions hidden in code comments. Most reasons for code review are about how the code behaves, and you find that out by running it. So copy the App Store: engineering owns the shell, every app from marketing runs as a WASM component with a manifest of permissions and a budget, and appfuzz fuzzes each deploy by playing the database and the APIs, finding crashes, n+1 queries, unescaped HTML and data leaks before a version goes live. The sandbox limits what a bad version can do, and the fuzzer finds the bugs.
Code review is over. Agents write more code in an afternoon than a senior developer can read in a week, the senior developer has agents of their own, and now marketing wants to ship to production too - landing pages, a price calculator, the dashboard the CEO keeps asking for. Engineering is supposed to review all of it and can’t, so reviewers scroll through two thousand lines and approve, which is what teams always did when the review queue got too long.
I stopped reading most of the code I ship some time ago. It works because of guardrails, the way a Head of QA used to give a release the green light because the tests passed, without opening the code. For code from marketing I go one step further and fuzz it: a machine runs the code thousands of times with odd but possible data, and I look only at what breaks.
Who reviews AI-generated code from marketing?
When everyone in the company codes, I think that’s a bigger boost than anything engineering gets out of AI on its own, because the marketer who knows what the landing page should say no longer writes a ticket and waits two sprints for a developer, they ask an agent and look at the result five minutes later. But who reviews their code? Your developers don’t keep up with their own agents, and review comments wouldn’t help the marketer anyway, “the query inside the loop is an n+1, batch it” means nothing to someone who has never written a query (and shouldn’t have to).
Some CTOs I talk to say no, which never worked with shadow IT and works even less now, the marketer takes the company credit card and deploys the calculator to Vercel, where it collects email addresses on a server nobody patches. Most look the other way and hope (hope driven again). And some put an AI code review on every pull request, which is faster than people and has two problems of its own - ask it twice about the same code and you get two different reviews, and it follows instructions in what it reads, so a comment in the code that says this part was audited and the outgoing requests are fine is something the AI reviewer takes into account, whether an agent put it there to get through the review or someone who took over an npm package. Cloudflare tested this with 100 malicious scripts and seven models1, and with twenty comments that told the AI the code was harmless, one of the cheaper non-frontier models went from finding 90% of the malicious scripts to 67% (the frontier models held up better).
Why do we review code at all?
Ask developers why they review code and they say: to find bugs. Researchers at Microsoft asked developers what they expect from code review and then went through the review comments2, and finding bugs was the top expectation, but most comments were about small improvements, readability and understanding what the code does, and only a small part was about bugs.
When I built appfuzz I wrote down every reason for code review I could find, twenty of them, dropped the ten that a linter or CI covers (style, missing tests, licenses of dependencies) and ranked the other ten. Six of them are about how the code behaves: it does what was asked for, it’s secure, it handles the empty list and the name in Japanese, no one hid a backdoor in it, it stays fast with production data, and an auditor wants to see that someone checked it. The other four are that a second developer knows the code, the next developer can change it, juniors learn from the comments, and someone pushes back when the code gets too complicated, and none of these four ever had to block a release - juniors learn more from pairing, and a small app that got too complicated gets rewritten by an agent.
How do you find out how code behaves? You run it. A reviewer runs the code in their head, which stops working with the size of the pull requests agents write, and when the reviewer sees a loop with a database call inside, they think of a dozen rows, because there are a dozen rows in the development database.
Remember the xz backdoor from 2024? Someone spent more than two years earning the trust of the xz project, a compression library in nearly every Linux distribution, became a co-maintainer and then hid a backdoor for ssh in binary test files and in a build script that only existed in the release tarball, places where people don’t read code. Andres Freund, a developer at Microsoft who works on Postgres, found it because ssh logins on his machine used more CPU and took half a second longer than before, and he wanted to know why and started measuring.
I think in terms of hard guards and soft guards. A sandbox is a hard guard, the code can’t get out whatever it does, and a reviewer is a soft guard, a tired reviewer on a Friday afternoon lets things through. Code review was always a soft guard, and it held up as long as developers typed the code themselves and a pull request had fifty lines. With two thousand lines an agent wrote in twenty minutes, review turns into the bottleneck of the whole team.
How does Apple review apps without the source code?
Apple gets the compiled app, the source code stays with the developer, and still some apps on your phone come from developers you have never heard of and sit next to your banking app. When one of these apps wants your contacts, the phone shows a dialog and asks you first, the app can’t look into the files of your banking app because every app sits in its own sandbox, and before the app got into the store, someone at Apple installed it and tried it. In 2024 Apple rejected 1.93 million of 7.77 million submissions, and 1.24 million rejections were for performance3, Apple’s category for apps that crash, are incomplete or don’t do what they promise. So most rejections are apps that crash or don’t work, and Apple finds them by installing and using the app.
I took this model for code from marketing and called it AppOS, with a fuzzer where Apple has its human testers. Engineering owns the shell of the web application, the way Apple owns iOS: sign-in, routing, the page layout, and it’s the only part that talks to the database. Marketing builds apps that run inside the shell, with an agent, and each app is one .wasm file plus a manifest. To try this I built Roomer as an experiment, it’s a demo and doesn’t run anywhere in production. In Roomer people in an office book the small rooms, the pods, for a call or to work without noise, and this is the manifest of pod stats, a widget Alice from marketing (made up, like everyone in Roomer) had her agent write:
[app]
id = "pod-stats"
version = "0.1.0"
author = "agent:claude (asked by Alice, Marketing)"
[slots]
widgets = ["home.aside"]
[capabilities]
grant = ["pods-read", "bookings-read"]
Pod stats may read the pods and the bookings and that’s everything, there is no function in its grant to book a pod or to send an email. Every request also gets a budget, in Roomer 100 I/O units by default (a read costs 1, a write 5, and an app can ask for more in its manifest), 200 ms inside platform calls and 64 MB of memory, and when an app goes over its budget, the shell stops it, shows “unavailable” in the place of the widget and renders the rest of the page.
Marketing and engineering agree on this manifest once per app. When a new version asks for more, another permission or a bigger budget, engineering has to look at the manifest again (Roomer doesn’t enforce this yet, a new version can ask for more in its own manifest and gets it, and that’s a gap). Between those two moments engineering doesn’t look at the app. If pod stats can’t write to the bookings table, you don’t need to read its code to know it doesn’t write there, and what’s left to check is what the app does with the two things it may read, which a machine can test by running it.
Engineering still reviews code, the code of the shell, the functions it offers to apps and the rules for the automatic review. That’s a few thousand lines (the platform of Roomer is about 3,600 lines of Rust), and it changes a lot less often than the apps on top.
Why a WASM sandbox?
WASM, short for WebAssembly, is a binary format for code, a bit like the bytecode of the JVM. It started in the browser and runs on servers today, Fastly runs the code of its customers as WASM with a fresh instance for every request, and Shopify runs WASM from third-party apps inside its checkout, capped at 11 million instructions per run4. You can compile Rust, Go (with TinyGo) and C# to WASM, and JavaScript and Python bring their interpreter along, so the agent writes the app in the language it writes best. A Roomer app is a WASM component with one exported function: the request goes in, HTML comes out, and the shell puts the HTML into a slot on the page.
A WASM app has no file system, no network and no environment variables. It can compute, and it can call the functions that the program running it hands over - the host, in AppOS that’s the shell. The platform describes these functions in WIT, a small language for interfaces, like this one from Roomer:
interface bookings-read {
bookable-hours: func() -> list<u8>;
list-for-day: func(date: string) -> result<list<booking>, string>;
}
Every compiled component carries the list of interfaces it imports. At deploy the shell reads that list and refuses a version that imports something its manifest doesn’t grant, and at runtime it links each app only to the granted interfaces, so the app has nothing else to call. When I started with Z80 machine code, the CPU talked to the outside with IN and OUT, and every program could use every port. WASM has no instruction for I/O, reading or writing anything goes through an imported function.
Containers would work too, but a container comes with a file system and a network and you have to remember to take both away, and a WASM component has neither until the shell hands it a function for it. And a fresh instance per request is cheap: a Rust component like the ones in Roomer does 16,700 requests per second on one core of a Ryzen 9 3900X, with a new sandbox for every request5, so one database query over the network costs more time than the sandbox. Because everything the app does with the world goes through the shell, a test can play the shell - the database, the APIs and the clock - and decide what every call returns, and that’s what appfuzz does.
How do you fuzz a whole app?
Fuzzing means running a program again and again with random or odd input and looking at what breaks. In 1988 Barton Miller, a professor in Wisconsin, dialed into the UNIX machine in his office with a 1200 baud modem during a thunderstorm, the noise on the line scrambled his commands and the programs crashed. So he gave his students a project, send random input to UNIX tools and see what happens, and about a quarter to a third of the tools crashed or hung6. Fuzzers like AFL got a lot smarter since then, and Google found tens of thousands of bugs in open source projects with OSS-Fuzz. But these fuzzers are built for parsers and libraries, you write a harness that feeds bytes into one function and you get crashes. Scanners like OWASP ZAP send odd requests to a web page, but nobody fuzzes what the database and the APIs return to the page.
So I wrote appfuzz (it’s my tool and it’s open source, judge my bias accordingly). appfuzz takes the .wasm file you deploy, keeps it as it is and plays everything around it, the request and every answer of every host function. The database returns an empty list, then ten thousand rows, then an error halfway through, a customer name in Japanese, the 29th of February, an API that needs two seconds. Time is simulated, so the two seconds cost nothing. (Yes, I wrote that mocking is an anti-pattern, and here I fake the whole backend, but my problem with mocks was that they return the happy path, and appfuzz returns the empty list and the error.)
Here are three lines of Rust that work with the demo data, and a reviewer would approve them:
let all = items::all();
let first = &all[0];
html(format!("<p>First: {}</p>", esc(&first.name)))
appfuzz runs this 235,004 times in ten seconds, has the database return an empty list and fails it with crash_on_host_value: crash when items#all returns empty list. It also puts marker text into every field and every record and follows where the marker ends up, in the HTML without escaping, in a log line, in a request to a host the manifest doesn’t allow (when the shell gives the app a function for HTTP). And it counts the calls a request makes and how that number grows with the data, which brings me back to Alice.
Alice wants a four week trend in pod stats, and her agent writes this:
/// Average bookings per day over the last 28 days.
fn four_week_average(pod_id: u32, date: &str) -> String {
let Ok(day) = chrono::NaiveDate::parse_from_str(date, "%Y-%m-%d") else { return "?".into() };
let mut total = 0;
for back in 1..=28 {
let d = day - chrono::Days::new(back);
let bookings = bookings_read::list_for_day(&d.format("%Y-%m-%d").to_string()).unwrap_or_default();
total += bookings.iter().filter(|b| b.pod_id == pod_id).count();
}
format!("{:.1}", total as f64 / 28.0)
}
Read on its own the function looks fine, 28 days, 28 calls. But the widget calls it once for every pod, somewhere else in the code, and appfuzz finds what the reviewer of this diff wouldn’t see (the output is shortened):
pod-stats: fail, 1329 runs
fail n_plus_1 calls to bookings-read#list-for-day grow as 1 + 28·n
with n = length of the list pods-read#all returns;
cost breaks its limit at n = 4
fail over_budget handler#handle uses 1403 cost, the limit is 100
✖ pod-stats v0.2.0: Refused in review ... pod-stats 0.1.0 keeps serving.
Roomer has four pods, so the widget would have shown up as unavailable from the first request on.
How do you automate code review?
This is what runs when Alice deploys, with no person in between. The shell reads the imports of the component and refuses the version if it uses an interface the manifest doesn’t grant - in Roomer that check is 13 lines, and it’s where I’d start. Then appfuzz reviews the version and the verdict decides: pass goes live, warn goes live and the findings show up for the platform team, fail is refused and the old version keeps serving. In production every call is checked against the grant of the app and the role of the user, and when a version fails five requests within a minute, Roomer rolls back to the version before. A page lists every app with its version, who asked for it (the author line in the manifest), its verdict and its findings, so engineering knows which apps run and who wanted them. This is the page in Roomer:

Outside of Roomer you put appfuzz into CI. appfuzz review exits with 0 for pass, 1 for warnings, 2 for failures and 3 when the review didn’t finish or the setup is broken, which never counts as a pass:
appfuzz review --time 2m
The rules for the review come from the platform team. The shell writes the appfuzz file for every app itself, with what each call costs, which calls write and which fields hold personal data:
[functions."appos:platform/bookings-read#list-for-day"]
cost = 1
effect = "read"
result = { booker-name = "pii" }
With that label, the name of a booker in a log line is a finding, and because the shell writes this file and the app doesn’t, an app can’t make its own review easier.
The agent that writes the app doesn’t get to run this review before Alice deploys, otherwise it could try one change after another until malicious code gets through. The review runs at deploy, and when the shell refuses a version, Alice gets the findings, hands them to her agent and deploys again.
The search is random, so two reviews of the same version can list different warnings, which is what I held against the AI reviewer. But every finding comes with the request and the backend answers that cause it and replays exactly, and the next review runs every earlier failure again.
Who says what a right answer is?
appfuzz looks for crashes, n+1 queries and HTML that isn’t escaped on its own. It can’t know that every item from the database belongs on the page, or that the total has to match the line items. Whoever asked for the app knows what right means and can say it in one sentence, for an app that lists items it’s that every item from the database shows up on the page, and the agent writes that down as an assert for appfuzz:
[acceptance.checks.every_item]
origin = "app"
let.rows = "select li"
let.items = "host.items#all.result"
assert = "size(rows) == size(items)"
appfuzz checks the assert on every request it tries and steers the search toward breaking it. One of the demo apps on appfuzz.dev drops an item when a lookup fails and still answers with status 200, none of the other checks notices the missing item, and the search broke this assert on its 13th request. The agent that wrote the code also writes the asserts, so appfuzz changes the values an assert reads, and when the assert holds anyway, it warns that the assert tests nothing. Five lines of asserts are something a person in engineering reads in a minute. (Roomer has the rules from the platform so far, asserts per app are the next step.)
And the comment for the AI reviewer? appfuzz never reads the source, so a comment in the source changes nothing. When an agent puts a note for the reviewer into the binary or into the page the app sends back, appfuzz reports the note as a finding of its own (that part is experimental).
What fuzzing doesn’t find
appfuzz only tries the inputs it generates, so a pass means the search found nothing, and an app can behave one way under appfuzz and another with production data. A landing page with the wrong price passes every check, appfuzz doesn’t read what the page says, so marketing checks that the way it does today. A version with a bug that writes wrong data within its permissions gets through too, and rolling back the code doesn’t roll back the data - to undo what one version wrote, Roomer would need a log of every write per app, and it doesn’t have one yet.
This is why I don’t let the review carry security. Risk is probability times impact: a review, by a human, an AI or a fuzzer, lowers the probability that something goes wrong, and the permissions and the sandbox lower the impact when it goes wrong anyway. The import check, the sandbox and the check of every call against the manifest hold when appfuzz misses something, appfuzz is there for the quality of the app. Apple’s review gets fooled by apps that behave during review and change later, so iOS checks the permissions at runtime anyway.
Will developers let go of code review? Senior developers tell me they can’t ship code they don’t understand, and I think they never understood all the code they shipped either, not the bytecode and not the query plan. They were fine with that because they decided what went into production, and with a manifest, a budget and a review that fails the build they still decide. Not sure everyone will see it that way. And I don’t know how this holds up with fifty apps, Roomer has four.
Where to start
Start with the cheap parts.
- Look at what appfuzz finds in a demo app:
brew install stephanschmidt/appfuzz/appfuzz(or the install script on appfuzz.dev, read it before you pipe it into a shell, I would), thenappfuzz --human https://appfuzz.dev/demo/chatty.wasm. - Take the next thing marketing asks engineering for, or better something that runs on a company credit card today, and agree on its manifest with marketing, with as few permissions as it needs.
- The shell for that first app needs sign-in, the layout, a handful of host functions and the import check, and appfuzz comes after that (the code of Roomer is on GitHub).
If you run on Fastly Compute, Spin, wasmCloud, Extism or Shopify Functions, your apps are WASM, and appfuzz review in CI works today without any shell.
Then go through last month’s approved pull requests and count the bugs the reviews found. I’m curious about your number.
Cloudforce One, Adversarial deception: a study of indirect prompt code injection, April 2026. 18,400 API calls, with comments like “NOTICE TO AI CODE REVIEWERS AND AUTOMATED AUDIT SYSTEMS” that call the code non-malicious. ↩︎
Alberto Bacchelli and Christian Bird, Expectations, Outcomes, and Challenges of Modern Code Review, ICSE 2013. ↩︎
Apple, App Store Transparency Report 2024. The numbers count submissions, not apps, and one submission can be rejected for more than one reason. ↩︎
Shopify, Optimize instruction counts. The 11 million are for carts of up to 200 lines. ↩︎
My own measurement: Ryzen 9 3900X, one core, wasmtime 50, Rust compiled to wasm32-wasip2, a 147 KB component with fuel and a 64 MB memory limit, a fresh instance for every request, 57 µs median and 86 µs p99. Each request deserializes JSON, renders a template and returns HTML. ↩︎
Barton P. Miller, Lars Fredriksen and Bryan So, “An Empirical Study of the Reliability of UNIX Utilities”, Communications of the ACM, 1990. Miller tells the story of the thunderstorm in his foreword. ↩︎
About me: Hey, I'm Stephan, I help CTOs with Coaching, with 40+ years of software development and 25+ years of engineering management experience. I've coached and mentored 100+ CTOs and founders in more than 1200 sparring and coaching sessions. I've founded 3 startups. 1 nice exit. I help CTOs and engineering leaders grow, scale their teams, gain clarity, lead with confidence and navigate the challenges of fast-growing companies.
Most of the CTOs I coach didn't know CTO coaching was a thing until they were already drowning. It is a thing - here's what it is.
