QualityPilot

AI Bug Detective

Auto-fix

Other test-quality tools tell you you have flaky tests. We open pull requests that fix them.

Status

Beta. Free plan: 0 fixes/day (paid feature). Pro: 5/day per repo. Team: 25/day. We're being deliberately conservative while we tune the model — your repo stays in your control.

How it works

  1. Your CI sends test runs via one of the QualityPilot reporters (jest / playwright / pytest).
  2. You enable auto-fix on a specific repo (default OFF on every repo).
  3. Hourly we scan the last 24h for failed tests on that repo. For each one not yet attempted, we fetch the source file from GitHub at the failing commit.
  4. Source goes through secret redaction before anything else.
  5. The redacted source + test name + framework go to gpt-4o-mini via OpenAI. The model returns either a proposed fix (with confidence 0..1) or noFixAvailable (e.g. flaky network, missing env var — not a code bug).
  6. If confidence is at or above your repo threshold (default 0.7, system minimum 0.5), we open a pull request from a stable branch qualitypilot/auto-fix/<hash>. Same fix re-running updates the existing PR; never opens duplicates.
  7. The PR carries labels qualitypilot/auto-fix, automated, and a confidence band, plus the LLM's reasoning in the body.
  8. You review and merge. We never automerge.

Secret redaction

Before any source code leaves QualityPilot for the LLM, it passes through a regex pass that scrubs:

  • Provider keys: OpenAI, Anthropic, Stripe, AWS access keys, GitHub PATs, npm tokens, JWTs, Slack tokens, Google API keys
  • Generic env-style assignments — SOMETHING_KEY = "..." with a long opaque value — keeps the variable name visible (LLM still sees "this is an API key being assigned") but redacts the value

This is defense in depth, not the only safeguard. The primary consent layer is opt-in per repo.

GitHub permissions

We use the OAuth access token already granted at sign-in (the same one that powers scans). No separate GitHub App. The token has repo scope so we can read source + open PRs on repos you have access to. We only ever push to branches matching qualitypilot/auto-fix/*. We never push to your default branch.

Cost + caps

  • Free: 0 fixes/day (auto-fix is a paid feature)
  • Pro: 5 fixes/day per repo
  • Team: 25/day
  • Enterprise: 100/day

You can tighten any cap further (never expand) on /dashboard/settings/auto-fix. Each fix attempt costs us roughly $0.0005 in OpenAI; caps protect you from a misbehaving CI generating thousands of failures and draining the budget.

What it does NOT do

  • Never auto-merges. Every PR requires your review.
  • Never modifies code unrelated to the failing test (the model is explicitly instructed; we validate on output).
  • Never invents APIs that aren't in the source it sees (same instruction).
  • Never runs on a repo without explicit opt-in.
  • Never fires above your confidence threshold — the model is asked to return its own honest confidence and we gate on it.
  • Never sends env files (.env is in our non-code-file filter — it never reaches the analyzer at all).

Disable

Toggle the per-repo switch off on /dashboard/settings/auto-fix. Existing in-flight jobs finish; no new ones queue. Already-opened PRs remain untouched (you can close or merge them as normal).

Get started

  1. Install a reporter in your test suite.
  2. Run your tests in CI with at least one failure (or wait for one to happen naturally).
  3. Enable auto-fix on the repo.
  4. Within an hour, expect a PR. Review it like any other PR.

Questions?

support@qlens.dev — security disclosures within 24 hours.