The Promise and Reality of AI Coding Agents

During my master's thesis at Frankfurt School, where I studied Artificial Intelligence and Data Science (MADS), I spent several months studying a question that felt both technical and surprisingly human: when an AI agent submits code to a real software project, how often does it actually get accepted by a reviewer?

To answer it, I analyzed more than 10,000 pull requests submitted autonomously by AI coding agents to popular open-source repositories. What I found was both encouraging and sobering.

A New Kind of Code Contributor

A pull request (PR) is how code changes get proposed in collaborative software projects. A developer writes code, submits a PR, and waits for teammates to review it before it gets merged into the main codebase. AI coding agents, such as Devin, GitHub Copilot, OpenAI Codex, Cursor, and Claude Code, now do this autonomously. They interpret a task, generate the code, and open a PR on their own.

The opportunity is real. Agents work around the clock. They handle repetitive tasks like fixing bugs, updating dependencies, and adding tests. They work far faster than any individual developer. For open-source projects and lean engineering teams alike, this represents a meaningful shift in how software gets built.

One in Three Gets Rejected

This is the number that surprised me most when I started exploring the data: roughly one in three agentic PRs get closed without being merged. A reviewer examined the AI-generated code and decided it was not ready to be included. In practice, that decision may come from a human reviewer, an automated pipeline, or a combination of both.

This does not mean AI agents are failing. The bar for integrating code into a real production project is high, and rightly so. But it does challenge the idea that agentic coding is a fully solved problem. The gatekeeping process is still very much in place, and it is not disappearing anytime soon.

Can We See Rejection Coming?

The core of my thesis involved training machine learning (ML) and natural language processing (NLP) models to predict, at the moment a PR is submitted, whether it would be rejected. I used the PR titles and descriptions, code changes, and various metadata as features for the predictive models.

It turned out to be a genuinely difficult problem. Even combining text embeddings, code representations, and structured metadata, the models performed only modestly above chance. More complex approaches, including large language models asked to reason about the PR, did not outperform the simpler classifier.

One practical implication stood out from this. The simplest and most computationally efficient ML model held up best. For teams thinking about building screening tools for agentic PRs, a lightweight classifier that runs cheaply at submission time may be more useful in practice than a sophisticated model that is expensive to operate and slower to adapt. Complexity did not pay off here.

Still, even the best-performing model faced a deeper challenge. Agent behavior shifts rapidly over time. A model trained on PRs from early in the year struggled to generalize to PRs from just a few months later. The agents themselves are evolving, and so are the projects they contribute to. This kind of drift makes it challenging for any static predictive system to keep pace.

The Review Process Still Matters

The practical takeaway is that the review process matters more, not less, as agent usage grows. If a third of agentic PRs are rejected and those rejections are difficult to anticipate in advance, software development teams need to stay actively engaged in the review process rather than simply rubber-stamp the output.

There is also an opportunity for the teams to build coding agents. If we better understand which signals correlate with rejection, such as the type of task, the size of the change, the programming language, or the specific agent, we can design systems that are more self-aware and more selective about the code and PRs that these agents submit.

Where Human Judgment Still Lives

Agentic software development is not a future trend. It is happening now, at scale, in real repositories. The opportunity is significant: AI coding agents enable faster iteration, broader access to contribution, and automation of the tedious parts of software work.

But the standards that determine whether code belongs to a project are still set by humans. The rules, the culture, the expectations of a codebase do not come from the agent. That, for now, remains distinctly human.