Concepts

The AI Bug Detective loop

The loop has four moves: ingest, detect, propose, act.

1. Ingest. Your CI sends test runs through one of our reporters. Each run includes per-case status, retries, durations, and the commit SHA.
2. Detect.A heuristic analyzer scores each failure. Did it fail intermittently? Did it start failing on a specific commit? Is the failure message similar to one we've seen before?
3. Propose. For high-signal failures the LLM reads the failing test source plus its imports and drafts a minimal patch. The patch is scored 0–10 for confidence.
4. Act. If the score clears your floor, we open a draft PR against the offending repo. You review, edit, merge — same as any human-authored PR.

Crucially, nothing in this loop auto-merges. The product opens PRs; humans close them.

Confidence threshold

Every proposed fix carries a confidence score from 0 to 10. The score blends signal quality (how clear is the failure?), context coverage (did we see enough source?), and patch shape (is the diff small, local, and obviously testable?).

Each repo has a confidence floor — fixes scoring below it are dropped silently. Defaults:

• Pro plan: floor of 7. Fewer PRs, higher-quality.
• Team plan: floor of 6. More throughput.
• You can raise the floor in repo settings; lowering it is gated.

Opt-in flow

Auto-fix is off by default in every repo. Turning it on is a deliberate three-step:

1. Install the QualityPilot GitHub App on the repo.
2. Open repo settings on /dashboard/settings and toggle auto-fix on.
3. Pick a daily cap (1, 5, or 25 PRs/day depending on plan). Below that we'll never spam your inbox.

We treat the opt-in as a contract: if you turn it off, in-flight proposals are cancelled and no further PRs are opened. The webhook keeps firing for visibility events but auto-fix actions stop immediately.

Multi-fix consolidation

Real CI fails in clusters. A flaky helper used by twelve tests will light up twelve failures, but it's one fix. The proposer deduplicates patches that touch the same lines and merges them into a single PR titled e.g. “Fix flaky timing in shared `waitForReady` helper”, listing all twelve affected tests in the PR body.

Consolidation is per-run. If two unrelated runs surface the same underlying issue back-to-back, you'll see two PRs. We don't try to merge across runs because the second run might have already validated the first fix.

Webhook architecture

QualityPilot sends webhooks for events you care about — fix proposed, fix opened, fix merged, fix rejected, run regressed. The payload envelope is stable:

POST https://your-receiver/qlens
X-QLens-Event: auto_fix.opened
X-QLens-Signature: sha256=...
Content-Type: application/json

{
  "id": "evt_01HX...",
  "type": "auto_fix.opened",
  "created_at": "2026-04-19T12:00:00Z",
  "data": {
    "repo": "owner/name",
    "pr_url": "https://github.com/owner/name/pull/123",
    "confidence": 8,
    "tests": ["src/foo.test.ts > flaky helper"]
  }
}

Receivers must verify the X-QLens-Signature header. We retry with exponential backoff for up to 24h on 5xx; 4xx responses are treated as permanent and not retried. The full HMAC recipe is on the security page.

Related reading

• Auto-fix — the operator-facing page on tuning the loop.
• Troubleshooting — what to do when something in the loop goes quiet.