GitLab regression testing

Run a test plan from GitLab CI after a deploy or as a blocking job.

Start from a test plan. GitLab CI calls the API to run that plan. It does not choose the tests. For a review of the merge request diff, use GitLab pull request testing.

Set it up

GitLab CI

Store the token as a CI/CD variable and call POST /v1/run.

Blocking mode

Poll the run and fail the job when the result is not passed.

Mobile build

Upload the build and run the plan against it.

1. Create an API token

   Go to Organization Settings → API Keys, create a key, and copy it when it is shown. It is not displayed again. Project-scoped keys are bound to one project. Org-scoped keys need projectShortId on API calls.

2. Store the token in GitLab

   Go to Settings → CI/CD → Variables and add:

   • Key: QA_TECH_API_TOKEN
   • Value: Your API token
   • Protected: ✅
   • Masked: ✅
3. Get the project short ID

   Copy proj_… from the project URL or Settings (for example proj_abc123). Org-scoped keys need this on POST /v1/run. See Understanding Different IDs.

4. Get the test plan short ID

   Open the plan under Test Plans and copy pln_… from the URL (for example pln_abc123). See Find the test plan short ID.

For complex payloads with dynamic values, write to a file first:

before_script:
  - |
    echo '{"key": "'$VARIABLE'"}' > /tmp/request.json
script:
  - curl ... --data @/tmp/request.json

What works on GitLab

Capability	GitLab
Review on a preview URL, before merge	Not this guide. A test plan run does not post a merge request review.
Review after merge, with no preview URL	Not this guide. See GitLab pull request testing.
Run a regression test plan from CI	Yes. POST /v1/run with the test plan short ID. You can override the URL for a preview or staging deploy.
Run that plan against a mobile build	Yes. Upload the build and pass applicationBuildShortId. See Mobile regression testing.
Block the pipeline until the plan finishes	Yes. Poll the run and fail the job when the result is not passed.
Schedule	On the test plan, with Manage Schedules. See Manual and scheduled runs.

What this does not do

• This path does not post a merge request review. Use GitLab pull request testing for that.
• GitLab cannot promote review tests into the suite. That agent exists on GitHub pull request testing only.
• Envoyer is not required. Use it only when the deploy itself is what should start the plan. See Envoyer.

Patterns

Basic setup

trigger_qatech:
  stage: test
  variables:
    QATECH_TEST_PLAN_SHORT_ID: 'pln_abc123'
  script:
    - >-
      jq -n
      --arg testPlanShortId "$QATECH_TEST_PLAN_SHORT_ID"
      --arg actor "$GITLAB_USER_LOGIN"
      --arg branch "$CI_COMMIT_REF_NAME"
      --arg commitHash "$CI_COMMIT_SHA"
      --arg repository "$CI_PROJECT_PATH"
      --arg repositoryUrl "$CI_PROJECT_URL"
      '{ trigger: "GITLAB", testPlanShortId: $testPlanShortId, actor: $actor, branch: $branch, commitHash: $commitHash, repository: $repository, repositoryUrl: $repositoryUrl }'
      > qatech-request.json
    - >-
      curl --fail-with-body
      --request POST
      --url "https://api.qa.tech/v1/run"
      --header "Authorization: Bearer $QA_TECH_API_TOKEN"
      --header "Content-Type: application/json"
      --data @qatech-request.json

Replace pln_abc123 with your test plan short ID (from your test plan page). The runner image must provide curl and jq.

The GITLAB trigger and repository metadata attribute the run to GitLab in QA.tech. The results page links the commit to $CI_PROJECT_URL/-/commit/$CI_COMMIT_SHA, including for self-hosted GitLab projects. Requests without this metadata remain generic API runs.

Run test plans on merge requests

test_mr:
  stage: test
  only:
    - merge_requests
  variables:
    QATECH_TEST_PLAN_SHORT_ID: 'pln-smoke-tests_abc123'
  script:
    - >-
      jq -n
      --arg testPlanShortId "$QATECH_TEST_PLAN_SHORT_ID"
      --arg actor "$GITLAB_USER_LOGIN"
      --arg branch "$CI_COMMIT_REF_NAME"
      --arg commitHash "$CI_COMMIT_SHA"
      --arg repository "$CI_PROJECT_PATH"
      --arg repositoryUrl "$CI_PROJECT_URL"
      '{ trigger: "GITLAB", testPlanShortId: $testPlanShortId, actor: $actor, branch: $branch, commitHash: $commitHash, repository: $repository, repositoryUrl: $repositoryUrl }'
      > qatech-request.json
    - >-
      curl --fail-with-body
      --request POST
      --url "https://api.qa.tech/v1/run"
      --header "Authorization: Bearer $QA_TECH_API_TOKEN"
      --header "Content-Type: application/json"
      --data @qatech-request.json

Test preview deployments via API

Pass dynamic URLs between jobs using dotenv artifacts:

stages:
  - deploy
  - test

deploy_preview:
  stage: deploy
  script:
    - echo "PREVIEW_URL=https://preview-${CI_MERGE_REQUEST_IID}.yourdomain.com" >> deploy.env
  artifacts:
    reports:
      dotenv: deploy.env

test_preview:
  stage: test
  dependencies:
    - deploy_preview
  before_script:
    - |
      echo '{"testPlanShortId":"pln-regression-suite_abc123","applications":[{"applicationShortId":"app-frontend_abc123","environment":{"url":"'$PREVIEW_URL'","name":"MR-'$CI_MERGE_REQUEST_IID'"}}]}' > /tmp/request.json
  script: >
    curl --request POST
    --url "https://api.qa.tech/v1/run"
    --header "Authorization: Bearer $QA_TECH_API_TOKEN"
    --header "Content-Type: application/json"
    --data @/tmp/request.json

The environment object also accepts optional customHeaders to persist auth or protection-bypass headers on that environment. Omit the field to leave stored headers unchanged; pass [] to clear them. Add devicePresetShortId on each application object in the same request. See Environment custom headers and Start Run API.

Mobile build

Native apps have no preview URL. Upload the APK or simulator .app, then pass applicationBuildShortId instead of url. How application builds work is on Mobile regression testing. The shared upload script is on API pull request testing.

test_mobile:
  stage: test
  only:
    - merge_requests
  script:
    - |
      # Upload the build first (see /pr-testing/api#upload-the-pr-build),
      # then start the plan against applicationBuildShortId:
      echo '{
        "trigger": "GITLAB",
        "testPlanShortId": "pln_abc123",
        "actor": "'"$GITLAB_USER_LOGIN"'",
        "branch": "'"$CI_COMMIT_REF_NAME"'",
        "commitHash": "'"$CI_COMMIT_SHA"'",
        "repository": "'"$CI_PROJECT_PATH"'",
        "repositoryUrl": "'"$CI_PROJECT_URL"'",
        "applications": [{
          "applicationShortId": "app_gXeBl2",
          "environment": {
            "applicationBuildShortId": "'"$BUILD_SHORT_ID"'"
          }
        }]
      }' > /tmp/request.json
    - >
      curl --fail-with-body
      --request POST
      --url "https://api.qa.tech/v1/run"
      --header "Authorization: Bearer $QA_TECH_API_TOKEN"
      --header "Content-Type: application/json"
      --data @/tmp/request.json

Replace pln_abc123, app_gXeBl2, and set BUILD_SHORT_ID from the upload response.

Blocking mode

Wait for test completion before proceeding with deployments:

trigger_qatech:
  stage: test
  script:
    # Start run and capture shortId
    - |
      RESPONSE=$(curl -s -X POST \
        "https://api.qa.tech/v1/run" \
        -H "Authorization: Bearer $QA_TECH_API_TOKEN" \
        -H "Content-Type: application/json" \
        -d "{\"testPlanShortId\": \"pln_abc123\"}")
      SHORT_ID=$(echo "$RESPONSE" | jq -r '.run.shortId')

    # Poll until completion (see Run Status API for details)
    - |
      while true; do
        RESPONSE=$(curl -s \
          "https://api.qa.tech/v1/run/$SHORT_ID" \
          -H "Authorization: Bearer $QA_TECH_API_TOKEN")
        STATUS=$(echo "$RESPONSE" | jq -r '.status')
        if [[ "$STATUS" == "COMPLETED" || "$STATUS" == "ERROR" || "$STATUS" == "CANCELLED" ]]; then
          RESULT=$(echo "$RESPONSE" | jq -r '.result')
          [[ "$RESULT" == "PASSED" ]] && exit 0 || exit 1
        fi
        sleep 30
      done

See Run Status API for polling logic details and error handling.

Custom Slack notifications

Override the notification channel for one run. See Per-run overrides.