GitHub App for PR Reviews

Automatic AI-powered pull request testing and reviews

Get autonomous AI-powered test coverage and reviews on every pull request. The QA.tech GitHub App uses AI exploratory testing to analyze code changes, create missing tests, and post comprehensive reviews based on test results.

Integration modes:

• AI exploratory (this page): Automatic AI-powered PR reviews - GitHub only
• API-driven: Manual test plan triggers via GitHub Actions or GitLab CI

See CI/CD Integration Overview to understand both modes and choose the right approach for your needs.

What It Does

Feature	Description
✅ Intelligent Test Selection	AI semantically matches PR changes to relevant tests (typically 5-15 tests selected)
✅ Gap-Only Test Generation	Creates 1-3 tests only when coverage gaps exist; most PRs create zero new tests
✅ Persistent Test Suite	Auto-generated tests become permanent regression tests for future PRs
✅ Preview Environment Testing	Tests against PR preview deployments
✅ Approval/Rejection	Posts reviews with pass/fail verdicts
❌ Manual Control	Fully autonomous - use GitHub Actions for manual test selection
❌ Backend-Only Testing	Requires UI access - can’t test microservices without frontend

How PR Reviews Work

PR opened/updated
  ↓
1. Classify Changes
   → User-facing? Continue
   → Docs/infra only? Skip testing, post info comment
  ↓
2. Assess Coverage
   → Find relevant existing tests
   → Identify gaps in coverage
  ↓
3. Create Tests (if needed)
   → Generate tests for untested functionality
   → Configure dependencies (e.g., login tests)
  ↓
4. Run Tests
   → Execute against PR preview environment
   → Wait for completion
  ↓
5. Post Review
   → ✅ Approve if all tests pass
   → ❌ Decline if tests fail
   → ℹ️ Informational if untestable

The agent posts reviews with:

• Verdict: Pass/fail/unable to verify
• What was tested: Description of coverage
• Results summary: Patterns and themes
• Test details: Table with individual test results

Reviews focus on test results only - no code quality opinions, implementation suggestions, or references to other bot comments.

Test Selection and Creation

How Tests Are Selected

The agent uses AI to semantically match PR changes to test goals - not keyword matching.

Example: PR changes checkout payment flow

Your project: 645 total tests

Agent analyzes:
├─ "Complete checkout with credit card" → RELEVANT ✓
├─ "Complete checkout with PayPal" → RELEVANT ✓  
├─ "Verify payment confirmation email" → RELEVANT ✓
├─ "User profile photo upload" → NOT RELEVANT ✗
└─ "Search products by category" → NOT RELEVANT ✗

Selected: 12 tests covering payment & checkout flows

Selection mechanics:

• Runs ALL tests it determines are relevant (no arbitrary limits)
• Only considers tests with status='enabled'
• More intelligent than running all tests every time

When Tests Are Created

Tests are created only when coverage gaps exist, such as when you’re developing a new feature not yet covered by your testing suite.

PR Type	Existing Tests Selected	New Tests Created	Total
Bug fix in login	3-7	0 (already covered)	3-7
Small feature	5-10	1-2 (fill gaps)	6-12
Major feature	15-25	3-5 (new functionality)	18-30
Refactor	10-20	0 (no new behavior)	10-20
Docs/infra only	0	0 (untestable via UI)	0

Example: PR adds Apple Pay to checkout

Agent assesses existing coverage:
├─ "Complete checkout with credit card" ✓ Exists
├─ "Complete checkout with PayPal" ✓ Exists
└─ Apple Pay integration ✗ Gap identified

Decision: Create 1 new test
→ "Complete checkout with Apple Pay"

Created tests:

• Stay in your suite labeled ‘ephemeral’
• Available for future PRs
• Manage in Settings → Test Cases (filter by ‘ephemeral’ to find them)

Example lifecycle:

PR #42: Add Apple Pay
├─ Agent creates "Complete checkout with Apple Pay"
├─ Test runs on PR #42 ✅
└─ Test persists with 'ephemeral' label

PR #58: Refactor checkout UI
├─ Agent finds existing test
├─ Runs it (no new test created) ✅
└─ Your suite now protects against Apple Pay regressions

The agent prevents duplicates by reading all existing tests first and using semantic deduplication. If one slips through, simply delete it in the UI.

Self-limiting: As your test suite grows, fewer tests are created automatically - better coverage means fewer gaps. Most PRs (bug fixes, refactors) create zero new tests.

Mobile and Responsive Testing: When a PR mentions mobile, tablet, or responsive design changes, the AI agent may automatically test with appropriate device presets. The agent detects keywords like “mobile”, “tablet”, “responsive”, or “Safari mobile” and can override device presets when creating or running tests. See Device Presets for configuring device settings.

How to Set It Up

1. Install GitHub App

   Go to Settings → Organization → Connections and add the GitHub App connection. Follow the OAuth flow to grant access to your repositories.

2. Configure Repository

   Navigate to Settings → Integrations and select the repository you want to enable PR reviews for.

   PR reviews are enabled automatically once you select a repository.

   Optional: Add review context to guide the agent (e.g., “Focus on security vulnerabilities” or “Validate accessibility standards”). This appears in the integration settings.

3. Configure Preview Deployments (if needed)

   If you use Vercel, Netlify, Render, Railway, or Fly.io, your preview deployments are detected automatically - skip to step 4.

   QA.tech automatically creates preview environment records in your project when it detects your preview deployments (from Vercel, Netlify, etc.). Your CI/CD platform handles the actual deployment - QA.tech just tests against the preview URLs. For more information, see Preview Environments.

   For CircleCI, Jenkins, GitLab CI, or custom CI/CD:

   QA.tech detects preview deployments using GitHub’s Deployments API. Your CI/CD needs to create deployment records after deploying.

   What QA.tech expects:

   1. GitHub deployment created for the PR commit SHA
   2. Deployment status set to success
   3. Status includes target_url field with your preview URL

   How to set this up:

   Your CI/CD makes two GitHub API calls after deployment:

   1. Create deployment with commit SHA and environment name
   2. Create deployment status with state: "success" and target_url: "https://your-preview-url.com"

   CI/CD platform docs for environment variables:

   • CircleCI Environment Variables - Use $CIRCLE_SHA1, $CIRCLE_PROJECT_USERNAME, $CIRCLE_PROJECT_REPONAME
   • GitLab CI/CD Variables - Use $CI_COMMIT_SHA, $CI_PROJECT_NAMESPACE, $CI_PROJECT_NAME
   • Jenkins Credentials - Use $GIT_COMMIT and custom variables
   • GitHub Actions - Use ${{ github.sha }}, ${{ github.repository_owner }}

   Need help? Contact [e-mail] with your CI/CD platform.

4. Map Environments (optional)

   If you have multiple Applications (e.g., frontend + backend), map GitHub deployment environments to the correct QA.tech Applications.

   How it works:

   1. Your CI/CD deploys a PR and creates a GitHub environment (e.g., “Preview” or “pr-123”)
   2. QA.tech detects the deployment
   3. Tests run using the mapped Application’s URL from that environment

   Location: Settings → Integrations → GitHub App → Map Environments

   Without GitHub deployments, QA.tech can’t test against PR-specific URLs and will use your default environment.

5. Create a Pull Request

   Once configured, the agent automatically runs on PRs:

   1. Detects code changes when PRs are opened or updated
   2. Determines which tests are relevant
   3. Creates new tests for untested functionality
   4. Runs all relevant tests against the PR preview
   5. Posts a review with approval or decline based on results

Common Questions

Will auto-generated tests pollute my test suite?

No. The agent only creates tests for coverage gaps - most PRs (bug fixes, refactors) create zero new tests. Even major features typically add 3-5 focused tests. The system is self-limiting: better coverage → fewer gaps → less generation.

Can I control which tests run?

The GitHub App is fully autonomous. For manual test selection, use GitHub Actions instead. You can use both: automatic PR reviews + manual deep testing on-demand.

PR reviews aren’t posting - what should I check?

• Verify GitHub App is installed and has repository access
• Check repository is configured in QA.tech Settings → Integrations
• Ensure PR has user-facing changes (docs/infra-only PRs are skipped)

Tests running against wrong URL?

• Map GitHub environments to Applications in Settings → Integrations → GitHub App
• Verify your CI/CD creates GitHub deployment records
• Check environment names match between GitHub and QA.tech

Agent created irrelevant tests?

• Add review context with specific guidelines in integration settings
• Update existing tests to better cover the functionality
• The agent learns from your existing test patterns

Related Documentation

• CI/CD Integration Overview - Learn about integration modes and capabilities
• GitHub Actions - API-driven testing with manual control over test plans
• Test Plans - Organize tests for API-driven runs