Web

End-to-End Testing Web Apps with Playwright and Azure DevOps

Web · Published 2026-10-27

Playwright drives real Chromium, Firefox and WebKit browsers to test a web app the way a user would. To run it in Azure DevOps, install the browsers on the build agent with npx playwright install --with-deps, run npx playwright test with a JUnit reporter, and publish the XML with PublishTestResults@2 so results appear in the pipeline's Tests tab, alongside the HTML report and traces as artifacts.

The test and configuration below were run with Playwright Test 1.64 on Node.js 22 against a small local app; the pipeline YAML uses standard Azure Pipelines tasks.

Set up Playwright

npm init playwright@latest

The initializer adds @playwright/test, a playwright.config.ts, an example test and, optionally, a GitHub Actions workflow you can ignore. Locally, npx playwright test runs the suite, --ui opens the interactive runner, and npx playwright codegen http://localhost:4173 records clicks as test code, a useful starting point that you then tidy up.

A first test with resilient locators

The app here is a minimal to-do list: a labelled text field, an "Add" button and a list.

import { test, expect } from '@playwright/test';

test('adds a todo item', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/Todo demo/);

  await page.getByLabel('New todo').fill('Write the pipeline');
  await page.getByRole('button', { name: 'Add' }).click();

  await expect(page.getByRole('listitem')).toHaveText(['Write the pipeline']);
  await expect(page.getByLabel('New todo')).toBeEmpty();
});

test('ignores empty input', async ({ page }) => {
  await page.goto('/');
  await page.getByRole('button', { name: 'Add' }).click();
  await expect(page.getByRole('listitem')).toHaveCount(0);
});

Three habits make Playwright suites stable:

• Locate by role, label and text (getByRole, getByLabel, getByText) rather than CSS classes or XPath. These match what users and assistive technologies see, survive styling changes and quietly enforce accessible markup. Add data-testid attributes only where no user-facing handle exists.
• Use web-first assertions such as await expect(locator).toHaveText(...). They retry until the condition holds or the timeout expires, so you never need waitForTimeout.
• Keep tests independent. Each test gets a fresh browser context (clean cookies and storage). Create the data a test needs through an API call or fixture, not by depending on a previous test.

If you come from Selenium, the auto-waiting is the biggest difference; the comparison in Scrapy vs Selenium explains why browser automation needs it.

A configuration that behaves differently in CI

Azure Pipelines sets the TF_BUILD variable, and you can define CI yourself. The config reads it to switch reporters, retries and parallelism:

import { defineConfig, devices } from '@playwright/test';

const isCI = !!process.env.CI;

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: isCI,                 // fail if a test.only slipped in
  retries: isCI ? 2 : 0,
  workers: isCI ? 1 : undefined,
  reporter: isCI
    ? [['junit', { outputFile: 'test-results/junit.xml' }], ['html', { open: 'never' }]]
    : 'list',
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:4173',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
  projects: [{ name: 'chromium', use: { ...devices['Desktop Chrome'] } }],
  webServer: process.env.BASE_URL
    ? undefined
    : { command: 'npm run preview -- --port 4173', url: 'http://localhost:4173', reuseExistingServer: !isCI },
});

Running 2 tests using 1 worker
··
  2 passed (3.1s)

Key choices:

• webServer starts the app before the tests and stops it afterwards. When BASE_URL is set, for example to a deployed staging site, the tests target that URL instead.
• retries with trace: 'on-first-retry' records a full trace (DOM snapshots, network, console) only when a test fails once, which keeps runs fast but gives you everything needed to debug a flaky failure.
• One worker in CI is a safe default on small hosted agents; raise it if your agents have more cores, or shard the suite across jobs with --shard=1/3.

The Azure DevOps pipeline

trigger:
  branches:
    include: [main]
pr:
  branches:
    include: [main]

pool:
  vmImage: ubuntu-latest

variables:
  CI: 'true'
  npm_config_cache: $(Pipeline.Workspace)/.npm

steps:
  - task: UseNode@1
    inputs:
      version: 22.x

  - task: Cache@2
    inputs:
      key: 'npm | "$(Agent.OS)" | package-lock.json'
      path: $(npm_config_cache)
    displayName: Cache npm

  - script: npm ci
    displayName: Install dependencies

  - script: npx playwright install --with-deps chromium
    displayName: Install Playwright browser

  - script: npx playwright test
    displayName: Run Playwright tests

  - task: PublishTestResults@2
    condition: succeededOrFailed()
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: test-results/junit.xml
      testRunTitle: Playwright e2e
      failTaskOnFailedTests: true

  - task: PublishPipelineArtifact@1
    condition: succeededOrFailed()
    inputs:
      targetPath: playwright-report
      artifact: playwright-report-$(System.JobAttempt)
    displayName: Publish HTML report

  - task: PublishPipelineArtifact@1
    condition: failed()
    inputs:
      targetPath: test-results
      artifact: test-results-$(System.JobAttempt)
    displayName: Publish traces and screenshots

Notes on the pipeline:

• --with-deps installs the operating system libraries the browsers need on the Ubuntu agent. Installing only chromium saves time if your config tests only one engine. The browser version must match the installed @playwright/test version, which is why the install runs after npm ci.
• condition: succeededOrFailed() publishes results even when tests fail, which is exactly when you need them.
• The Tests tab shows each test with its outcome and history, so flaky tests become visible across runs.
• Traces land in test-results. Download the artifact and open a trace with npx playwright show-trace path/to/trace.zip, or drop it on the Playwright trace viewer page.
• The artifact name includes $(System.JobAttempt) because a re-run job cannot publish an artifact with a name that already exists.

As an alternative to installing browsers on every run, you can run the job in Microsoft's official Playwright Docker image (mcr.microsoft.com/playwright, tagged with the matching version) using a container job.

Testing a deployed environment

End-to-end tests are most valuable against the real deployment. Split the pipeline into stages: build and deploy to staging, then run the Playwright stage with BASE_URL set to the staging URL, and only promote to production if it passes. If the app is hosted on Azure Static Web Apps, its preview environments give each pull request its own URL; see deploying a static web app from an Azure DevOps pipeline. Keep credentials for test accounts in secret variables and map them explicitly into the test step with env:, because secret variables are not exposed to scripts automatically.

What to test end to end

Browser tests are slower than unit tests. Reserve them for critical journeys: sign-up and login, the main create-read-update-delete flow of your app (see what a CRUD application is), checkout or payment hand-off, and anything that broke in production before. Push edge cases down to unit and API tests.

Summary

Write Playwright tests with role and label locators and web-first assertions, make the config CI-aware (JUnit reporter, retries with traces, forbidOnly), install browsers with --with-deps on the agent, and publish the JUnit XML with PublishTestResults@2 and the HTML report and traces as pipeline artifacts, even when tests fail.

Sources

• Playwright documentation, Continuous Integration and Test configuration.
• Playwright documentation, Reporters.
• Microsoft Learn, PublishTestResults@2 task reference.

Read next

More in Web

Web

Scrapy vs Selenium

Scrapy vs Selenium compared: architecture, JavaScript rendering, speed, scaling and pipelines, with working code for each and when to use scrapy-playwright.

Web

What is a CRUD application?

CRUD means Create, Read, Update, Delete. See how it maps to SQL and REST HTTP methods, with a runnable Python example, validation, soft deletes and idempotency.

Web

HTTP requests with urllib3

How to send GET and POST requests with urllib3: PoolManager, query fields, JSON bodies, headers, timeouts, Retry, redirects, TLS, streaming and errors (v2).