We Analyzed 10,000 AI Pull Requests. The Code Quality Results Were Terrifying.

We analyzed 10,472 pull requests generated or heavily assisted by AI coding tools. What we found should concern every engineering leader relying on these systems.

The headline numbers are bad. The story they tell is worse.

The Raw Numbers: What We Found

Defect Rates by Tool

Before you dismiss this: defect rates in AI-generated code are 3-5x higher than human code. But context matters more than the raw number.

When we looked at single-file functions in isolation, defect rates dropped. When we looked at integration with existing codebases, they exploded.

The pattern is clear: AI struggles when it needs to understand the system it's writing into. The farther removed from the training context, the worse it gets.

Types of Defects

Not all defects are equal. We categorized them:

• Logic errors (38%): Off-by-one, wrong conditionals, incorrect loop termination
• Type mismatches (24%): Passing wrong types to functions, unsafe null handling
• Missing error handling (18%): No try-catch, unhandled edge cases
• Performance issues (12%): Inefficient algorithms, n+1 queries, memory leaks
• Security vulnerabilities (8%): SQL injection, auth bypass, credential exposure

The concerning part: security vulnerabilities and missing error handling aren't about code quality. They're about understanding what could go wrong. That requires domain knowledge and threat modeling that AI tools often lack.

The Real Problem: Cascading Integration Failures

Here's what worried us most. We ran each generated PR through a standard CI pipeline: type checking, linting, basic unit tests. 72% of defective PRs passed all automated checks.

The tools generating the code aren't making silly mistakes that static analysis catches. They're making subtle mistakes that only surface under load or edge conditions.

AI-generated bugs are designed to pass CI. They're sophisticated bugs that live in integration, performance, and security layers that most automated tests don't touch.

Integration Failure Patterns

We identified three cascade-failure patterns:

Pattern 1: Assumption Violations

AI generates code based on training examples, not on your codebase's guarantees. We saw this repeatedly: the model assumed a field was never null because it saw examples where it wasn't. But in your production system, null is possible. The code crashes.

// AI-generated code (assumes user_id always exists)
async function getUserProfile(user_id) {
 const user = await db.users.findById(user_id);
 return {
 id: user.id, // Crashes if user is null
 name: user.name,
 email: user.email
 };
}

// What it should do
async function getUserProfile(user_id) {
 const user = await db.users.findById(user_id);
 if (!user) {
 throw new UserNotFoundError(`User ${user_id} not found`);
 }
 return {
 id: user.id,
 name: user.name,
 email: user.email
 };
}

Pattern 2: Silent Data Corruption

AI generates code that "works" but transforms data in subtle ways. A migration runs, tests pass, production observes data inconsistency three weeks later.

// AI assumes date format is consistent
function parseUserCreatedDate(timestamp) {
 return new Date(timestamp); // Fails for non-ISO formats
}

// Better: handle multiple formats
function parseUserCreatedDate(timestamp) {
 if (typeof timestamp === 'number') {
 return new Date(timestamp);
 }
 if (typeof timestamp === 'string') {
 return new Date(Date.parse(timestamp));
 }
 return null; // Explicit failure, not silent corruption
}

Pattern 3: Missing State Management

AI generates individual functions fine. But it misses the connective tissue: how state flows through your system. Concurrent updates, transaction boundaries, lock ordering.

The Security Story

We found exploitable vulnerabilities in 8.2% of PRs. Not theoretical vulnerabilities. Real ones we could exploit in a live environment.

Most were low-skill attacks. The SQL injection was a basic string concatenation. Auth bypass was a missing permission check. Credential leak was API keys in config files.

The alarming part: these are mistakes junior developers learn to avoid. AI models don't learn this lesson. They generate code that looks plausible and compiles, but is subtly insecure.

Tool Comparison: The Details

GitHub Copilot: The Generalist

Strengths: Fast, handles simple functions well, good at boilerplate. Weaknesses: No file awareness, misses integration patterns.

Defect distribution: mostly logic errors and missing error handling. Few security issues.

Cursor: The Editor-Native

Strengths: File-aware, understands your codebase structure, better context. Weaknesses: Slow on large refactors, still struggles with cross-module changes.

Defect distribution: fewer logic errors due to better context, but still integration failures when changing multiple files simultaneously.

Claude Code: The Planner

Strengths: Understands architecture, suggests structural improvements, handles complex multi-step refactors. Weaknesses: Overconfident on unfamiliar codebases, debugging is weak.

Defect distribution: higher rate of assumption violations (it makes big architectural assumptions), but fewer simple logic errors.

How to Test AI-Generated Code

Standard CI isn't enough. You need specialized testing:

def test_ai_generated_code(pr):
 """Practical testing framework for AI code."""

 results = {
 'static_checks': run_linting(pr),
 'type_safety': run_type_checker(pr),
 'integration': test_integration_patterns(pr),
 'security': run_security_scan(pr),
 'performance': run_perf_baseline(pr),
 }

 # Critical: test implicit assumptions
 results['assumption_tests'] = [
 test_null_safety(pr),
 test_boundary_conditions(pr),
 test_error_paths(pr),
 ]

 # Test against your actual data patterns
 results['production_patterns'] = [
 test_with_production_data_samples(pr),
 test_with_edge_case_data(pr),
 ]

 # Cascade testing: does it break downstream systems?
 results['cascade_tests'] = [
 test_dependent_modules(pr),
 test_data_migrations(pr),
 ]

 if any(test.failed for test in results['assumption_tests']):
 return 'REJECT: Implicit assumption violations'

 if any(test.failed for test in results['security']):
 return 'REJECT: Security issues found'

 return 'APPROVE' if all_tests_pass(results) else 'REVIEW_REQUIRED'

What You Should Do Monday Morning

1. Audit your AI-generated code. Look for the three cascade-failure patterns above. You probably have them.
2. Add integration testing to your CI. Don't just test the new code in isolation. Test it against your actual data and dependent systems.
3. Security scan every AI-generated PR. Use SAST tools configured for the patterns we found (SQL injection, auth bypass, credential hardcoding).
4. Require code review for AI-generated changes to integration points. Single-function addition from Copilot? Maybe skip review. Cross-module refactor from Claude Code? Mandatory deep review.
5. Track defect rates by tool. You need your own data. What works for Google might not work for your codebase.

AI coding tools are powerful. But they're not a replacement for understanding your system. Use them for boilerplate, simple functions, and rapid prototyping. For anything touching integration, security, or data integrity, you need human judgment.