Would You Pay $1 for Agentic Code Review on PRs?

It might surprise people that I’m investing time in LLM tooling — “Agentic” tooling even — because I’ve always been somewhat of an AI-skeptic. But, this code-review project is not my first foray into the category; I am also working on a task runner called camas that optionally includes an MCP and skills, and indeed it was used for the development of code-review. In turn camas now benefits from code-reviews’s agentic review on every PR.

Perhaps I am more of an “AI-industry” skeptic, or wary of hype and fear-mongering, than I am skeptical of LLMs themselves. I am not convinced that “AI agents” can replace engineers, or any occupation, really. I can only speak for my experience as a software engineer — the frontier models like Opus and Fable 5 are particularly well-suited for writing software, and on their own, I don’t think they’re that good at it. But I’m not sticking my head in the sand either. My experience is that LLMs, when bounded by expert judgement and deterministic checks, provide an incredible, albeit hard to quantify, boost to the pace of development and testing. One category where I’m very happy with their utility is code review, particularly agentic code review.

Lately, I’ve started to dread what a post-AI-bubble or post-token-subsidy world would look like. Will models that provide affordable inference survive? How will Open Source Software (OSS) projects benefit from LLMs if contributors are no longer able to use subsidized flat-rate plans?

#Agentic Code Review

Some definitions are in order since “agentic” is a bit of a buzz word.

#Peer Code Review

Let’s relate everything back to what code review means to you. In order to review the code, you must first read the code — additions and removals — and you may choose to also interact with the code. Interacting with the code would require copying (pulling) the changes to your local development environment and then fiddling with it: running the test suite, running the library or application, modifying code as you do so, and in my line of work — something that an LLM agent cannot do — flashing firmware to a physical device and interacting with it. The process uses aspects of engineering, design, quality assurance (QA), testing, and writing.

The kind of review that is read-only — an audit of changes — is like a “Static LLM Code Review”. A review that interacts with — or even modifies — the code is like an “Agentic LLM Code Review”.

#Static LLM Code Review

This is a relatively cheap approach to automated code review because it only reads. It’s also easy to implement: just pipe your git diff into an API chat prompt and ask for a review. There are many examples of ready made GitHub Actions to do this and it’s simple enough that you can do it yourself (something that will be a theme in this post.)

Some may claim that this is agentic because the chatbot could do things like make web searches to validate claims. Nonsense: those are other aspects of a standard read-only code review.

Still others might call an LLM doing this sort of review “static analysis”, to which I disagree fervently. Static analysis is something that software engineers are already familiar with and it is a deterministic routine to assess some qualities of the software. LLMs are stochastic not deterministic so they cannot perform what we traditionally consider static analysis…

#Agentic LLM Code Review

…unless those LLMs are given access to the deterministic tools that do perform static analysis. This is the realm of Agentic Code Review and it’s something that I have been quite impressed by ever since my first use of Claude Code’s absolute token incinerator of a command: /code-review.

An agentic code review uses a harness (like Claude Code) and can run compilers, linters, launch parallel subagents in worktrees, modify the code to confirm hypotheses, research dependencies, check for CVEs, possibly even interact with the application (though not on physical hardware for us firmware engineers 🤖).

Where a static LLM code review may come back in a few minutes after consuming less than 1M tokens, an agentic review may go on for tens of minutes, or even an hour, using 5M tokens and beyond. Which is why you’ll see Anthropic advertise this service as costing $15-$25 per PR. And frankly, I could see this going out to $100 eventually, based on the hour+ /code-review sessions I’ve run locally.

In my experience, these agentic reviews reliably surface real, sometimes quite subtle, findings. And that’s what we want. But how can we enable agentic code review at an affordable cost for OSS?

#Open Source Models

tl;dr it’s DeepSeek v4

I’m not going to claim to understand API token pricing, but what I will claim is that as long as DeepSeek is offering the rate they do on input cache hits, DeepSeek v4 Pro/Flash are an incredible value for this kind of open source agentic code review and it’s what I’ve been experimenting with for the last couple of weeks.

Here is an actual cost table from an actual agentic code review on an actual PR: $0.20 for a 14m 54s run.

DeepSeek has no charge for Cache write (AFAIK, please correct me if I’m wrong) and an incredible discount on DeepSeek v4 Pro Cache read of $0.003625/MTok, leading to an affordable review for open source.

#Wait… where’s the API key? How is it safe from forks? And how was that code review produced?

Basically I wanted this thing, and in the era of practically free inference on frontier models, I will admit that the implementation and spec is written by an LLM.

Wait, no, don’t leave!

Because there is not a precedent for agentic code review on public repos, not to mention implementation of granular time and cost constraints, I decided that the first step was to produce a Minimum Viable Product (MVP) and set it loose in the wild. If I had worked on it without LLM assistance, I’d still be drafting the spec and validating GitHub Actions security boundaries instead of publishing a write up.

#code-review

#What the MVP reference implementation accomplishes

• Mitigates risk of leaking your API token by triage, network egress lock, and structured rendering (but you do need to use a burner key with a small amount on it, to be safe).
• CI-reactive review paths:
  • On CI success, launches the “full review”.
  • On CI failure, launches a minimal “mechanic” review meant to guide contributors to correct mistakes. Optional, can just do no review on CI failure to avoid sinking cost.
• Structured review output:
  • Each review will meet requirements because it is generated from markdown templates, not an LLM’s whims.
  • The LLM delivers review in a JSON document that has been schema validated.
  • LLMs reading the review are directed to that JSON document, not GH API + markdown parsing.
• Deterministic(-ish!) time and cost limits on the agentic review. This was by far the most difficult aspect of development and stands to be improved — harness improvements would have the biggest impact here.
• Start from a GH comment like /code-review $0.25 Make sure to check that all docs were updated
  • Limit the ability to start a code review to trusted repository members.
  • Control costs, time limits, and review focus for iterative PR phases.
• Full control:
  • change the harness
  • change the models
  • change the prompts
  • change the supporting tooling
  • change the provided context (feed it the CI test reports, for example)
• Incremental re-reviews: the system reads previous review JSONs to jump-start the review context
• Traceability — the entire review transcript is uploaded

All of that is accomplished with markdown templates, the reference review-reusable.yaml, and the code-review helper tool.

Calling 5K lines of TS with 10K lines of supporting tests a “helper” might mean I’m finally vibe-pilled 🤦‍♀️.

#You Can Try it Now

1. Create a review.yaml in your GitHub repo that imports code-review’s review-reusable.yaml and sets the options.
2. Add .github/prices.json to get cost estimates for your reviews.
3. Add the MODEL_API_KEY secret to your repo (your deepseek platform API key).
4. Once this is present on main, your next PR into main will trigger the review workflow.

#The spec

So that’s the reference implementation that anyone can try out today, but the core of the repository is the idea itself, characterized in this spec.md document. I hope that this proof-of-concept inspires others to work in this category. And of course contribution to code-review is welcome!

I don’t know how the market-side could work, and I don’t know how much inference is going to cost in the long run, but personally I would share my open source projects as training data in exchange for reduced token rates.

Here are some highlights of what the specification document defines and how the MVP went about realizing them.

#GitHub Actions & Public Forks

Briefly, there are real security vulnerabilities created by allowing public forks to open a pull request that triggers github actions.

There are two primary concerns in the case of enabling agentic code review.

1. Secret exfiltration. You at least need to have an API token as a GitHub secret, so that’s a likely target. Open a PR that modifies the GitHub workflow to print the secret to the log, then use it to pay for inference.
2. Spoofing. Using our API token to ask the LLM to do some malicious behavior, or just plain stealing your inference for themselves.

In GitHub terms, the way that code-review mitigates risk is by running the workflow from main, not from the untrusted fork. This means that in the review context, the repository source code is unmodified by the PR’s changes. So how do we we run the review of the PR if we’re on main, not the branch corresponding to the PR?

1. Checkout main.
2. Install our code-review helper tool (described above) and Claude Code from npm.
3. Run the users own environment setup steps.
4. Lock network egress to a short trusted list.
5. Run the security triage: a Claude Code agent evaluates the diff of the PR, with a tiny subset of tools available, and determines whether the PR is safe for a full agentic review, as true or false. This is not without risk, which is why it’s important for open source repositories to use API keys with small amounts on them — $5.00 — so that API key leakage is not a catastrophe. I am inviting everyone to try to break it on my own repositories by opening PRs against them with prompt injections.
6. Run the agentic review, branching on the result of CI. You could always just not even start this review workflow if CI fails. Or you can run a $0.05 review to help a new contributor understand why CI failed for them.
7. Validate the JSON schema of the review.
8. Generate the review comments from the JSON + templates and post them to the PR.
9. Upload the review JSON and full transcripts.

I’m not pretending it’s simple. It’s a 700+ line YAML file and that’s after abstracting lots of stuff in the code-review helper app. But let’s jump to the spelled out threat model so that we’re not GitHub-centric.

#Threat Model

The most important part of the specification is the threat model and suggestions for mitigation.

#Assets

There are the assets shared with anyone who creates a PR from a fork.

• Model API key — spends money; the primary target.
• Write credential — can modify the repository or its conversation; the secondary target.
• Reviewer runtime — transient, but its environment holds the model key.

#Threats

#Conclusion

I will continue to use code-review on my own OSS projects since I’ve already seen it catch issues that slipped by local review and my own (“static”) review. The maintenance workflow for it tends to be pretty simple to manage:

• Notice a problem with a review? Point an agent to the review and ask it to create an issue, including links to the comment and run, on the code-review repo.
• Assign one or more issues to an agent that will create a PR on code-review.
• Instruct that agent to wait on code-review’s automated CI + code reviews, and iterate.

Looking forward to seeing more users and bug reports, or new systems based on the threat mitigations demonstrated. Good luck!