CI/CD

Why the CLI is CI-friendly

The testsprite CLI is designed from the ground up for unattended execution:

Non-interactive auth — set TESTSPRITE_API_KEY as a secret; the CLI never prompts when the variable is present.
Stable --output json — every command emits a machine-readable JSON envelope that your pipeline can parse and forward.
Stable exit codes — exit 0 means the run passed; any non-zero exit lets the shell fail the job. No text scraping required.

Your target must be reachable from TestSprite’s cloud. Tests run on TestSprite’s cloud runners, so --target-url has to be a public http(s) address. localhost, 127.0.0.1, ::1, 0.0.0.0, link-local, and private (RFC 1918) addresses are rejected during pre-flight with exit code 5, before any run is triggered — so a dev server started inside the CI job cannot be tested directly. This is the most common first-run CI failure.In CI, point --target-url at a publicly reachable preview, staging, or deploy-preview URL (for example, the per-pull-request preview URL your hosting provider creates). To test a local app, use the TestSprite MCP Server, which opens a tunnel for you — the CLI itself does not tunnel.

Authenticating in CI

Set your API key as a repository or environment secret. In GitHub, go to Settings → Secrets and variables → Actions and click New repository secret:

GitHub Actions secrets page with the New repository secret button

Then expose it to the job:

env:
  TESTSPRITE_API_KEY: ${{ secrets.TESTSPRITE_API_KEY }}

The CLI resolves TESTSPRITE_API_KEY before the credentials file, so no testsprite setup step is needed in CI.

Important: Never inline an API key in a workflow file or a script committed to version control. Always source it from a secret store.

In most pipelines the env var alone is enough — you don’t need to run any setup command. If you do need a credentials file on disk (for other tooling), write one non-interactively from the same env var with --no-agent to skip the skill install:

testsprite setup --from-env --yes --no-agent

This reads TESTSPRITE_API_KEY from the environment, verifies the key, writes ~/.testsprite/credentials, and prints an identity summary.

Add testsprite doctor as a preflight step to catch a missing key, a bad endpoint, or an unreachable API before the real test step runs. It exits non-zero only on a genuine failure — warnings like a missing skill install don’t fail the step:

testsprite doctor && testsprite test run --all --project "$PROJECT_ID" --wait

Gating on the result

The CLI exits non-zero whenever a run does not pass — failed, blocked, cancelled, or timed out. Your shell or CI runner treats that as a job failure automatically.

testsprite test run test_3a9f21c7 --wait --output json > result.json
# exit 0 = passed; any other exit fails the step

On failure you can pull the artifact before the job ends:

testsprite test run test_3a9f21c7 --wait --output json > result.json || \
  testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure

A GitHub Actions workflow

Install and invoke the testsprite CLI directly, as the example below does — this is the supported way to run TestSprite in CI, on any platform. If you previously set up an older TestSprite GitHub Action from the marketplace, migrating to the CLI gives you stable exit codes, JUnit output, and a consistent pass/fail verdict.

The example below runs all backend tests in a project on every pull request and fails the job if any test does not pass:

name: TestSprite

on:
  pull_request:

jobs:
  testsprite:
    runs-on: ubuntu-latest
    env:
      TESTSPRITE_API_KEY: ${{ secrets.TESTSPRITE_API_KEY }}
      PROJECT_ID: proj_8f0f6

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install TestSprite CLI
        run: npm install -g @testsprite/testsprite-cli

      - name: Run tests
        run: |
          testsprite test run --all --project "$PROJECT_ID" \
            --wait --output json > result.json

      - name: Download failure artifacts
        if: failure()
        run: |
          testsprite test failure get \
            "$(jq -r '.run.testId' result.json)" \
            --out ./.testsprite/failure

      - name: Upload failure bundle
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: testsprite-failure
          path: ./.testsprite/failure

--all runs every test in the project in wave order — producers first, teardown last — so backend dependencies are always satisfied. Keep each project to a single test type (frontend or backend, set at project create); don’t mix both in one project. To target one test explicitly, pass its test-id, or loop over testsprite test list --project "$PROJECT_ID" --output json.

JUnit XML reports

For CI systems that render JUnit natively (GitHub Actions test reporters, GitLab, Jenkins), add --report junit --report-file <path> to a batch, --wait run or rerun. The CLI writes the XML sidecar atomically once polling finishes — --output json on stdout is unchanged, so you can use both together. Extending the “Run tests” step from the workflow above:

      - name: Run tests
        run: |
          testsprite test run --all --project "$PROJECT_ID" \
            --wait --output json \
            --report junit --report-file ./results.xml > result.json

      - name: Publish test report
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: TestSprite
          path: results.xml
          reporter: java-junit

--report is gated to the batch --wait path:

Available on test run --all --wait and on a batch test rerun --wait (--all, or several test-ids...) — not on a single-test run or rerun.
Requires --wait (the report is written after polling completes) and --report-file <path>.
--report-suite-name <name> optionally overrides the <testsuite name="..."> attribute; it defaults to testsprite:<projectId>.
Only junit is accepted for --report today.

GitHub-native annotations and summaries

On GitHub Actions (GITHUB_ACTIONS=true — set automatically on every runner), a --wait run needs no extra flags to become CI-native:

Every non-passed run emits one ::error:: workflow-command line, so failures annotate the PR checks tab with the test id, status, and dashboard link.
A Markdown results table is appended to the job summary ($GITHUB_STEP_SUMMARY).

This works on test run <test-id> --wait, test run --all --wait, and a batch test rerun --wait. Two optional flags extend it:

--summary-file <path> — also write a reduced machine summary JSON, {total, passed, failed, timedOut, runs[]}, for downstream steps that don’t want to parse the full --output json stream.
--gh-output — force the annotations outside GitHub Actions, so you can preview the exact CI output locally.

Everything is written even when the command exits non-zero — the annotations and summary land before the exit-code gate, so a red build still gets its explanation. All writes are best-effort: a failed write never changes the exit code. Tests that never dispatched (rate-deferred, conflicted, not found) appear as non-passed rows, so a partial batch can’t read as all-passed.

      - name: Run tests
        run: |
          testsprite test run --all --project "$PROJECT_ID" \
            --wait --output json \
            --summary-file ./summary.json > result.json
        # No reporter action needed: failures annotate the PR automatically,
        # and the results table appears on the workflow run's Summary page.

Parsing JSON output

Pipe --output json to jq to extract fields for downstream steps:

testsprite test run test_3a9f21c7 --wait --output json \
  | jq '.run.status'

"passed"

Read the dashboard link for a PR comment or Slack notification:

testsprite test run test_3a9f21c7 --wait --output json \
  | jq -r '.run.dashboardUrl'

The top-level run object contains status, runId, startedAt, finishedAt, codeVersion, and dashboardUrl (when the backend returns a portal link).

Rate limits and retries

The server caps run-triggers at 60 per minute per key. The CLI throttles itself to 50 per minute and automatically retries RATE_LIMITED responses, honoring the Retry-After header (capped at 3 retries). For large batches, tune the in-flight concurrency:

testsprite test run --all --project "$PROJECT_ID" \
  --wait --max-concurrency 20

For slow networks or a slow first request, increase the per-request timeout:

testsprite test run test_3a9f21c7 --wait \
  --request-timeout 300

Reuse an --idempotency-key to make retries safe — if the server already accepted the request with that key, it returns the original response instead of triggering a duplicate run:

testsprite test run test_3a9f21c7 --wait \
  --idempotency-key "ci-run-${{ github.run_id }}-${{ github.run_attempt }}"

Handling exit codes

The exit code is your primary signal. Two codes deserve special handling in a pipeline:

Code	Meaning	Recommended action
`11`	Rate limited	Honor `Retry-After`; the CLI retries automatically, but if you hit it outside a run command, back off and retry
`12`	Insufficient credits	Non-retriable — top up your balance before re-queuing the job

All other non-zero exits (auth error, test not found, network failure, test failed) should fail the job and surface the artifact.

Exit Codes

The complete exit-code table

Where to Go Next

Coding Agent Integration

Let your agent drive the verification loop without a human in the middle

Running Tests

All flags and modes for triggering and waiting on runs

Exit Codes

Full exit-code table for branching your pipeline

Configuration

Env vars, profiles, and per-request timeouts

Getting Started

Core

Integrations

Reference

Troubleshooting

Why the CLI is CI-friendly

Authenticating in CI

Gating on the result

A GitHub Actions workflow

JUnit XML reports

GitHub-native annotations and summaries

Parsing JSON output

Rate limits and retries

Handling exit codes

Exit Codes

Where to Go Next

Coding Agent Integration

Running Tests

Exit Codes

Configuration

​Why the CLI is CI-friendly

​Authenticating in CI

​Gating on the result

​A GitHub Actions workflow

​JUnit XML reports

​GitHub-native annotations and summaries

​Parsing JSON output

​Rate limits and retries

​Handling exit codes

Exit Codes

​Where to Go Next

Coding Agent Integration

Running Tests

Exit Codes

Configuration

Why the CLI is CI-friendly

Authenticating in CI

Gating on the result

A GitHub Actions workflow

JUnit XML reports

GitHub-native annotations and summaries

Parsing JSON output

Rate limits and retries

Handling exit codes

Where to Go Next