GitHub regression testing

Run a test plan from GitHub Actions after a deploy or as a blocking check.

Start from a test plan. GitHub Actions runs that plan. It does not choose the tests. For a review of the pull request diff, use GitHub pull request testing.

Set it up

Test Run Action

Add the secret and a workflow that runs the plan.

Mobile build

Upload the build and run the plan against it.

The post-merge agent is separate. After a reviewed pull request merges, it can promote tests from that review into the suite. It does not start the test plan.

1. Configure Secrets

   Add a secret to your GitHub repository (Settings → Secrets and variables → Actions). Create the token in Organization Settings → API Keys. Store it as QATECH_API_TOKEN. Project-scoped keys are bound to one project. Org-scoped keys can see multiple projects and need projectShortId on API calls.

2. Get the project short ID

   Copy proj_… from the project URL or Settings (for example proj_abc123). Pass it as project_short_id on the action. See Understanding Different IDs.

3. Get the test plan short ID

   Open the plan under Test Plans and copy pln_… from the URL (for example pln_abc123). See Find the test plan short ID.

4. Create Workflow

   Create .github/workflows/qatech.yml:

   name: QA.tech Tests
   on:
     push:
       branches: [main]
   
   jobs:
     test:
       runs-on: ubuntu-latest
       steps:
         - uses: QAdottech/run-action@v4
           with:
             project_short_id: 'proj_abc123'
             api_token: ${{ secrets.QATECH_API_TOKEN }}
             test_plan_short_id: 'pln_abc123'
             blocking: true
   

What works on GitHub

Capability	GitHub
Review on a preview URL, before merge	Not this guide. The Test Run Action does not post a pull request review.
Review after merge, with no preview URL	Not this guide. See GitHub pull request testing.
Run a regression test plan from CI	Yes. The Test Run Action runs the plan you name, including against a preview URL you pass in.
Run that plan against a mobile build	Yes. Upload the build and pass applicationBuildShortId. See Mobile regression testing.
Block the workflow until the plan finishes	Yes. Set blocking on the Test Run Action.
Schedule	On the test plan, with Manage Schedules. See Manual and scheduled runs.

What this does not do

• The Test Run Action does not post a pull request review and does not pick tests from the diff.
• Envoyer is a different trigger for the same kind of plan. It is not required when you have GitHub Actions. See Envoyer.

Patterns

Run on pull requests

name: PR Tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'smoke-tests'
          blocking: true

Test preview deployments

name: Test Preview
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      preview_url: ${{ steps.deploy.outputs.url }}
    steps:
      - name: Deploy to Vercel
        id: deploy
        run: |
          # Your deployment logic
          echo "url=https://preview-${{ github.event.pull_request.number }}.vercel.app" >> $GITHUB_OUTPUT

  test:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'regression-suite'
          blocking: true
          applications_config: |
            {
              "applications": {
                "frontend-app": {
                  "environment": {
                    "url": "${{ needs.deploy.outputs.preview_url }}",
                    "name": "PR-${{ github.event.pull_request.number }}",
                    "customHeaders": [
                      {
                        "domains": ["*.vercel.app"],
                        "headers": {
                          "x-vercel-protection-bypass": "${{ secrets.VERCEL_AUTOMATION_BYPASS_SECRET }}",
                          "x-vercel-set-bypass-cookie": "true"
                        }
                      }
                    ]
                  }
                }
              }
            }

This pattern passes the preview URL into the Test Run Action. It does not create GitHub deployment records. If you also want the GitHub App to pick up the same preview for automatic PR reviews, add the steps in GitHub Deployments.

Test a mobile pull request build

Native apps have no preview URL. Build the APK or simulator .app in the workflow, upload it, then pass applicationBuildShortId in applications_config instead of url. How application builds work is on Mobile regression testing. The shared upload script is on API pull request testing.

name: QA.tech mobile PR tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build APK
        run: ./gradlew assembleDebug

      - name: Upload build to QA.tech
        id: upload
        env:
          QATECH_API_TOKEN: ${{ secrets.QATECH_API_TOKEN }}
          APK_PATH: app/build/outputs/apk/debug/app-debug.apk
        run: |
          APP_ID="app_gXeBl2"
          FILE_NAME=$(basename "$APK_PATH")
          UPLOAD_RESPONSE=$(curl -sSf -X POST "https://api.qa.tech/v1/applications/$APP_ID/builds/upload-url" \
            -H "Authorization: Bearer $QATECH_API_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"fileName\": \"$FILE_NAME\"}")
          UPLOAD_URL=$(echo "$UPLOAD_RESPONSE" | jq -r '.uploadUrl')
          BUILD_TOKEN=$(echo "$UPLOAD_RESPONSE" | jq -r '.buildToken')
          curl -sSf -X PUT "$UPLOAD_URL" \
            --upload-file "$APK_PATH" \
            -H "Content-Type: application/octet-stream"
          BUILD_RESPONSE=$(curl -sSf -X POST "https://api.qa.tech/v1/applications/$APP_ID/builds" \
            -H "Authorization: Bearer $QATECH_API_TOKEN" \
            -H "Content-Type: application/json" \
            -d "{\"platform\": \"android\", \"buildToken\": \"$BUILD_TOKEN\"}")
          echo "build_short_id=$(echo "$BUILD_RESPONSE" | jq -r '.applicationBuildShortId')" >> "$GITHUB_OUTPUT"

      - uses: QAdottech/run-action@v4
        with:
          project_short_id: 'proj_abc123'
          api_token: ${{ secrets.QATECH_API_TOKEN }}
          test_plan_short_id: 'pln_abc123'
          blocking: true
          applications_config: |
            {
              "applications": {
                "app_gXeBl2": {
                  "environment": {
                    "applicationBuildShortId": "${{ steps.upload.outputs.build_short_id }}"
                  }
                }
              }
            }

Replace app_gXeBl2, pln_abc123, proj_abc123, and the APK path. Set blocking: true if the workflow should fail when tests fail.

Native Android and iOS examples assume android/ or an .app already exist. Expo and React Native managed apps gitignore those folders. In CI, generate them before upload:

- run: npm ci
- run: npx expo prebuild --platform android --non-interactive --no-install
- run: cd android && ./gradlew assembleDebug --no-daemon

The debug APK path is typically android/app/build/outputs/apk/debug/app-debug.apk. EAS preview APKs also work if you wait for the build and download the artifact. Still upload a simulator or emulator binary, not a store .ipa. Full build steps are in Preparing Your App Build.

iOS builds need a macOS runner and a simulator .app. Zip it, use "platform": "ios", and point BUILD_FILE at the archive. Full xcodebuild flags are in Mobile App Testing.

Persist environment custom headers

Pass customHeaders on any environment in applications_config to persist host-pattern header rules (auth and protection bypass). The Test Run Action and Change Review Action both accept this field. The preview deployment example shows it in a full workflow.

{
  "url": "https://preview.example.com",
  "name": "PR-123",
  "customHeaders": [
    {
      "domains": ["*.preview.example.com"],
      "headers": {
        "x-vercel-protection-bypass": "YOUR_SECRET",
        "x-vercel-set-bypass-cookie": "true"
      }
    }
  ]
}

customHeaders works with url, shortId, or applicationBuildShortId. Omit the field to leave stored headers unchanged. Pass [] to clear them. The Start Run API uses the same customHeaders shape on applications[].environment (array of application objects, not a map). See Environment custom headers.

Use action outputs

- uses: QAdottech/run-action@v4
  id: qatech
  with:
    project_short_id: 'proj_abc123'
    api_token: ${{ secrets.QATECH_API_TOKEN }}
    blocking: true

- name: Check Results
  if: steps.qatech.outputs.run_result == 'FAILED'
  run: echo "Tests failed! See ${{ steps.qatech.outputs.run_url }}"

Test Run Action reference

Inputs

Input	Description	Required	Default
project_short_id	QA.tech project short ID (for example proj_abc123)	Yes	-
api_token	QA.tech API token	Yes	-
test_plan_short_id	Test plan short ID to run	No	All tests
blocking	Wait for test results before completing	No	false
applications_config	JSON with application environment and device preset overrides. Optional customHeaders on each environment persist auth/bypass headers; see Environment custom headers.	No	-
api_url	Custom API URL	No	https://api.qa.tech

Outputs

Output	Description
run_created	Whether the test run was created successfully
run_short_id	The short ID of the run
run_url	The URL of the run
run_status	Final status (COMPLETED, ERROR, CANCELLED, or TIMED_OUT) - only when blocking: true
run_result	Test result (PASSED, FAILED, SKIPPED) - only when blocking: true