Skip to main content
Use this when the change is an Azure Repos pull request. To run a test plan you already have, use Azure DevOps regression testing.

Set it up

Pull request review

Azure Pipelines, the API, or an @qa.tech comment starts the review.

After merge

No preview environment. Review the merged pull request after the deploy step.

What works on Azure DevOps

What this does not do

  • Azure DevOps has no deployment event QA.tech can wait on. A review starts from Azure Pipelines, the API, or an @qa.tech comment.
  • Prepare change review accepts GitHub pull request URLs. An Azure DevOps review starts when you send the preview URL.
  • The triage fields on the change review API apply to GitHub. On Azure DevOps, triage follows the repository toggle.
  • The post-merge agent updates the suite. It does not run the GitHub retro that reads comment reactions.
  • This connection is for Azure DevOps Services (https://dev.azure.com/{organization} or https://{organization}.visualstudio.com).
What a review does on every CI is on How a review works.

Connect Azure DevOps

Azure DevOps has no installable app. QA.tech uses an organization URL and a personal access token. Comments are posted as the user who created the token.
Create the token on a dedicated Azure DevOps user, not a personal account. A personal token stops working when that user is removed, and every comment shows that user’s name.
1

Create a personal access token

In Azure DevOps, create a token with Code (Read & write). That scope lets QA.tech read the repository and post pull request comments, including screenshots.
2

Connect the organization

Open Settings → Integrations → Azure DevOps. Enter the organization URL, such as https://dev.azure.com/your-organization, and the token. The connection is shared by every project in the organization.
3

Add a repository

On the same page, add each repository QA.tech should review. Pick the applications those pull requests are tested against. With none selected, reviews use the project default application.
4

Choose how reviews start

Review Trigger defaults to Azure Pipelines. API is for a script you already run. Manually starts a review only from an @qa.tech comment. A comment that mentions @qa.tech starts or re-runs a review for every trigger.
5

Subscribe to service hooks

Saving the repository shows Service hook setup: a URL, the username qatech, and a basic auth secret. In the Azure DevOps project, open Project settings → Service hooks, add a Web Hooks subscription for each event below, and paste those three values. Azure DevOps does not sign the payload, so that secret is what authenticates the delivery. Azure DevOps webhook documentation covers the form in their UI.Subscribe to:
  • Pull request created
  • Pull request updated
  • Pull request merge attempted
  • Pull request commented on
Pull request commented on is what delivers an @qa.tech mention. Pull request merge attempted is what tells QA.tech a pull request merged, which the post-merge agent needs.
Each repository needs its own service hooks. The URL and secret appear after the repository is added. Optional Review Instructions on the repository guide every review of that repository, up to 1024 characters. For rules that apply to every repository, use PR Review Rules in project knowledge.

Start a review from Azure Pipelines

Put this step in the same job as the deploy step, after that step has set DEPLOY_URL to this pull request’s preview URL. Azure DevOps service hooks never include that URL.
In the deploy step, publish the preview URL into DEPLOY_URL:
$(System.CollectionUri) already ends with a slash. The JSON body is single-quoted so Azure Pipelines expands $(...) before the shell runs. $QATECH_API_TOKEN is a shell variable and stays in double quotes. Secret pipeline variables are not exposed to scripts on their own, which is why the step maps the token through env.
  • Create QATECH_API_TOKEN as a secret pipeline variable. The value is a QA.tech API key from Organization Settings → API Keys.
  • Replace proj_abc123 and app_gXeBl2 with your project and application short IDs. An organization-scoped key needs projectShortId. A project-scoped key may omit it.
  • Keep curl -sSf so a rejected request fails the step. The step does not wait for the review to finish.
  • If the deploy step is in another job, pass DEPLOY_URL across jobs with an output variable.
  • The repository settings include Copy agent prompt when the trigger is Azure Pipelines. That prompt contains this step with your project’s short IDs filled in.
The same request is documented on Start change review chat. Send commitHash as $(Build.SourceVersion) so the review is tied to the commit the pipeline built.

Start a review from the API

Use this when something other than the pipeline step above already knows the preview URL. The repository still has to be connected, with service hooks, so QA.tech can read the diff and post the review.
prUrl is the pull request URL: https://dev.azure.com/{organization}/{project}/_git/{repository}/pullrequest/{id}. The same path works on {organization}.visualstudio.com. Optional postMerge (enabled, and an optional prompt of up to 4000 characters) overrides the repository’s post-merge settings for that pull request. See Post-merge agent.

Start a review from a comment

Comment on the pull request and mention @qa.tech. This needs the Pull request commented on service hook. Add instructions (@qa.tech test the checkout flow) or a preview URL to test a specific environment. With no URL, the review uses the application’s configured environment. QA.tech replies with a link to the conversation. If the organization cannot start a run, that reply says so and the review does not start.

What QA.tech posts

The finished review is one comment on the pull request. A later review of the same pull request updates that comment instead of adding another. Screenshots in the review are uploaded to the pull request, because Azure DevOps only renders images from its own host. The comment is authored by the user who created the personal access token.

Code access

Code Access is on for a new repository. Chat and reviews can clone the repository with the connection’s token and search its code. Turn it off on the repository to stop cloning. With Code Access on, Logic Verification (beta) can model rule-heavy changes (dates, money, permissions, limits) from source and check them with a solver. That can add a few minutes to those reviews.

PR triage

PR triage is on for a new repository. QA.tech skips pull requests that cannot be tested with end-to-end or API tests, and does not post a review comment for those. It needs the pull request service hooks, because triage reads the pull request ingested from them. The skip rules are the ones on PR triage. Azure DevOps does not post a status when a pull request is skipped. Comment @qa.tech to run the review anyway. The API triage override is for GitHub pull requests. On Azure DevOps, use the repository’s PR Triage toggle and PR triage prompt.

Post-merge agent

Post-merge agent is off until you enable it on the repository. After a pull request merges, the agent keeps the test suite up to date. It only does what the prompt asks. It needs the Pull request merge attempted service hook, and a completed review of that pull request. Example prompts are on GitHub’s post-merge agent. Azure DevOps uses the same agent and does not run the GitHub retro afterward. postMerge on a change review request overrides enabled and prompt for that pull request. Omit it to use the repository settings.

After merge

When pull requests have no preview environment, run the review after the merge lands and the deploy finishes. The agent reads the merged diff and tests staging or production. The result is posted on the merged pull request. Before you add the step:
  • The repository is connected, with the pull request service hooks, so the agent can read the diff and post the result.
  • Staging exists in Settings → Applications & Envs, and you have its environment short ID (for example env_aB3xY9).
  • QA.tech can reach staging. If staging is private or behind a VPN, see Environment access.
Place this step after the deploy step. It runs on builds that are not pull request validations. A default Azure Repos merge commit starts with Merged PR 123:. If your merge commits do not include that line, the step skips them.
The agent tests the URL saved on that environment in QA.tech. When the staging URL changes, update the environment, not the pipeline. To test a URL that is not saved as an environment, send environment.url instead of environment.shortId. ubuntu-latest includes curl and jq. The token is a secret pipeline variable. The other env entries pass predefined variables into the script without putting them inside the JSON by macro expansion.