CI/CD Integration

CLI Runner

FastTest Agent includes a dedicated CI runner (fasttest-ci) that executes test suites headlessly — no MCP server or AI assistant needed.

npx -y @fasttest-ai/qa-agent ci \
  --api-key $FASTTEST_API_KEY \
  --suite-id $QA_SUITE_ID

Exit codes: The CLI runner exits with code 0 when all tests pass, and 1 when any test fails or an error occurs. CI pipelines will automatically fail the build on test failures.

All CLI flags:

Flag	Description
--api-key <key>	Organization API key (required)
--suite-id <id>	Test suite ID to run (required)
--base-url <url>	Cloud API URL (default: https://api.fasttest.ai)
--app-url <url>	Override the app base URL for this run
--environment <name>	Environment name (e.g., staging, production)
--pr-url <url>	GitHub PR URL to post results as a comment
--browser <type>	chromium, firefox, or webkit (default: chromium)
--test-case-ids <ids>	Comma-separated test case IDs to run (default: all)
--json	Output results as JSON (see below)

JSON Output

Pass --json to get machine-readable output for custom CI integrations:

npx -y @fasttest-ai/qa-agent ci \
  --api-key $FASTTEST_API_KEY \
  --suite-id $QA_SUITE_ID \
  --json

The JSON output includes the execution ID, overall status, and per-test-case results with pass/fail, duration, and any healing that occurred. The exit code is still 0 for pass and 1 for failure.

Auto-Generated CI Workflow

FastTest Agent can generate a ready-to-use GitHub Actions workflow and shell command for any test suite. From the dashboard, go to Settings → CI/CD, select a project and suite, and click Generate.

The generator:

• Detects all {{VAR_NAME}} placeholders in your test suite
• Lists the required environment variables with descriptions
• Outputs a GitHub Actions YAML and a standalone shell command

You can also generate via the REST API:

curl -X POST https://api.fasttest.ai/api/v1/qa/ci/generate \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"suite_id": "your-suite-id"}'

GitHub PR Comments

FastTest Agent can post test results directly on pull requests. When you run tests with a PR URL, results appear as a comment — including pass/fail counts, timing, regressions, fixes, and any selectors that were auto-healed.

From your AI assistant:

Run the login suite against this PR: https://github.com/myorg/myrepo/pull/42

Or use the run tool with the pr_url parameter:

{
  "suite_name": "Login Flow",
  "pr_url": "https://github.com/myorg/myrepo/pull/42"
}

Setting Up GitHub Integration

1. Create a GitHub Personal Access Token

   Go to GitHub → Settings → Developer Settings → Personal Access Tokens and create a token:

   • Classic token: Select the repo scope.
   • Fine-grained token: Grant Contents: Read and Pull requests: Read and write permissions for the target repositories.

2. Store the token

   Ask your AI assistant:

   Set my GitHub token to ghp_xxxxxxxxxxxx

   The token is encrypted and stored with your organization. All team members share the same token.

3. Run tests with a PR URL

   Include the pr_url parameter when running tests. Results are posted as a PR comment automatically.

Tip

The PR comment is updated on each run — it doesn’t create duplicate comments. It shows the latest results with a timestamp.

Running Tests from CI (REST API)

You can trigger test runs from CI pipelines by calling the REST API directly.

Authentication

All API requests require an x-api-key header:

curl -H "x-api-key: YOUR_API_KEY" \
  https://api.fasttest.ai/api/v1/qa/projects/

Start a Test Run

curl -X POST https://api.fasttest.ai/api/v1/qa/execution/run \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "suite_id": "your-suite-id",
    "browser": "chromium"
  }'

Response:

{
  "execution_id": "exec-789",
  "status": "running",
  "test_cases": [...]
}

Check Execution Status

curl https://api.fasttest.ai/api/v1/qa/execution/executions/exec-789 \
  -H "x-api-key: YOUR_API_KEY"

Post Results to a PR

curl -X POST https://api.fasttest.ai/api/v1/qa/github/pr-comment \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "pr_url": "https://github.com/owner/repo/pull/42",
    "execution_id": "exec-789"
  }'

Note

For the full API reference, visit the interactive docs at https://api.fasttest.ai/docs (Swagger UI).

Example: GitHub Actions with CLI Runner

The recommended approach uses the fasttest-ci CLI runner:

name: FastTest Agent Tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Install Playwright browsers
        run: npx playwright install chromium --with-deps

      - name: Run FastTest Agent tests
        env:
          FASTTEST_API_KEY: ${{ secrets.FASTTEST_API_KEY }}
          TEST_USER_EMAIL: ${{ secrets.TEST_USER_EMAIL }}
          TEST_USER_PASSWORD: ${{ secrets.TEST_USER_PASSWORD }}
        run: |
          npx -y @fasttest-ai/qa-agent ci \
            --api-key $FASTTEST_API_KEY \
            --suite-id ${{ vars.QA_SUITE_ID }} \
            --pr-url ${{ github.event.pull_request.html_url }}

Set FASTTEST_API_KEY and any {{VAR_NAME}} secrets as repository secrets. Set QA_SUITE_ID as a repository variable.

Tip

FastTest Agent tracks required_variables for each suite — the list of {{VAR_NAME}} placeholders used in test steps. Check the suite details in the dashboard or use list(type='suites') to see what environment variables you need to set in CI.

Example: GitHub Actions with REST API

If you prefer the REST API approach:

name: FastTest Agent Tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Run FastTest Agent tests
        run: |
          RESPONSE=$(curl -s -X POST https://api.fasttest.ai/api/v1/qa/execution/run \
            -H "x-api-key: ${{ secrets.FASTTEST_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{"suite_id": "${{ vars.QA_SUITE_ID }}"}')

          EXEC_ID=$(echo $RESPONSE | jq -r '.execution_id')

          # Wait for completion (poll every 10s)
          for i in $(seq 1 30); do
            STATUS=$(curl -s https://api.fasttest.ai/api/v1/qa/execution/executions/$EXEC_ID \
              -H "x-api-key: ${{ secrets.FASTTEST_API_KEY }}" | jq -r '.status')
            if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then break; fi
            sleep 10
          done

          # Post results to PR
          curl -X POST https://api.fasttest.ai/api/v1/qa/github/pr-comment \
            -H "x-api-key: ${{ secrets.FASTTEST_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d "{\"pr_url\": \"${{ github.event.pull_request.html_url }}\", \"execution_id\": \"$EXEC_ID\"}"