# Key Terms Source: https://docs.testsprite.com/cli/concepts/key-terms Core CLI concepts — projects, tests, runs, statuses, credits, scopes, and failure bundles — all in one place. This page defines the building blocks you'll see across every `testsprite` command. If a term in the docs feels unfamiliar, this is the place to land. ## Project A **project** is the top-level named container for your tests. Frontend projects target a live public URL; backend projects point at a codebase. Every project has a stable `projectId` you use in most commands. A project has: * **Type**: `frontend` or `backend` * **Target**: a public `https://` URL (frontend) or a codebase (backend) * **Tests**: every test case you have created inside it * **Run history**: every execution of those tests, across all surfaces Frontend projects must use a public, non-localhost URL. The CLI validates this before sending anything to the cloud. You reference a project by `projectId` whenever you list tests, trigger a run, or create a new test: ```bash theme={null} testsprite project list testsprite test list --project proj_8f0f6 testsprite test run test_3a9f21c7 --wait ``` Create a project from the CLI with `project create`, or from the Web Portal — both write to the same backend. ```bash theme={null} testsprite project create --type frontend --name "Checkout App" --url https://app.example.com ``` ## Test A **test** lives inside a project and is the unit the CLI runs, reruns, and reads results from. Every test has a type, a status, and a run history. A test has: * **Type**: `frontend` (an ordered plan of browser steps) or `backend` (executable test code, typically Python) * **Status**: one of the nine normalized values in the [Status](#status) table below * **Run history**: every time the test was executed, across CLI, Portal, MCP, and scheduled runs * **Test ID**: a stable `testId` used with `test get`, `test run`, `test failure get`, and more Frontend tests describe intent as a list of steps (`planSteps[]`). Backend tests carry executable code (typically Python) with optional variable dependencies (`--produces` / `--needs`). Read a test's current state: ```bash theme={null} testsprite test get test_3a9f21c7 testsprite test result test_3a9f21c7 ``` Run it and wait for a verdict: ```bash theme={null} testsprite test run test_3a9f21c7 --wait ``` Pull the failure bundle when it does not pass: ```bash theme={null} testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure ``` ## Run and Run ID A **run** is one execution of a test, from trigger to terminal verdict. Every run is identified by a stable `runId` minted by the backend before the cloud test engine starts. A run captures: * **Verdict**: one of `passed`, `failed`, `blocked`, or `cancelled` * **Steps**: a log of every step executed during that run * **Artifacts**: DOM snapshots (rendered as text), root-cause hypothesis, recommended fix target — all tied to a single `snapshotId` * **Source**: which surface triggered it — `cli`, `portal`, `mcp`, `schedule`, or `github_action` Runs are durable. A concurrent Portal or scheduled run never overwrites your run's artifacts. The CLI surfaces the `runId` at trigger time and after `--wait`. Use it to resume a poll or pin the failure bundle to exactly that run: ```bash theme={null} # Resume waiting for a run that was still in flight testsprite test wait run_5c1d... # Pin the failure bundle to one specific run (immune to later runs) testsprite test artifact get run_5c1d... --out ./.testsprite/runs/run_5c1d ``` Compare this to `test failure get `, which always returns the *latest* failure — useful but mutable if runs overlap. ## Status **Status** is the normalized state of a test or a run. The CLI displays it in text mode and includes it in `--output json` responses. Use status values to filter lists and drive CI branching logic. ### Test status (9 values) | Value | Meaning | Terminal? | | :------------------- | :----------------------------------------------------------- | :-------- | | draft | Test exists but has no executable code yet | No | | ready | Has code or a plan, never been run | No | | queued | Run accepted; cloud engine not yet started | No | | running | Execution in flight (pre-exec, exec, or post-exec analysis) | No | | passed | Latest completed run passed | Yes | | failed | Latest completed run failed (including infrastructure crash) | Yes | | blocked | Run rejected before a real verdict was reached | Yes | | cancelled | Run was cancelled before a verdict | Yes | | unknown | Status cannot be derived | — | Terminal statuses are the states the CLI considers "done" when polling with `--wait`. Exit 0 means passed; exit 1 means failed, blocked, or cancelled. ### Run status (6 values) Run-level status is a subset: queued, running, passed, failed, blocked, cancelled. Terminal run statuses are the same four: passed, failed, blocked, cancelled. Filter a test's run history by status or other dimensions: ```bash theme={null} testsprite test list --project proj_8f0f6 --status failed,blocked testsprite test result test_3a9f21c7 --history --source cli --since 7d ``` ## Run Source Every run is tagged with the surface that triggered it. The **run source** lets you filter history and understand where a pass or failure came from. The five source values are: | Source | When it appears | | :------------------------ | :-------------------------------------------------------------- | | cli | Any `testsprite test run` or `testsprite test rerun` invocation | | portal | A manual run from the Web Portal dashboard | | mcp | A run triggered by the MCP Server plugin in your IDE | | schedule | A scheduled run configured in the Portal | | github\_action | A run triggered from a GitHub Actions workflow | All CLI-triggered runs — including reruns — carry `source: "cli"`. Filter run history to only CLI runs over the last 7 days: ```bash theme={null} testsprite test result test_3a9f21c7 --history --source cli --since 7d ``` This is useful when a Portal or scheduled run has also fired recently and you want to isolate your agent's invocations. ## Credits **Credits** are the consumption unit for test execution. The CLI reports your balance via `testsprite usage` and exits non-zero when balance is insufficient. Key rules: * A fresh `testsprite test run` **charges credits**. * A `testsprite test rerun` is billed the same as a fresh run (0.5 credits frontend / 0.2 credits backend); legacy V2 accounts: a clean verbatim FE replay remains free. * Auto-heal is on by default and uses a small amount of credit only when it actually repairs a step — see [Rerun & Auto-Heal](/cli/core/rerun-and-auto-heal#auto-heal). * Backend test runs consume credits too, billed per your plan — the exact per-run cost isn't exposed by the API, so check `testsprite usage` or the pricing page before a large batch. * Insufficient balance → the CLI exits with a non-zero code. See [Exit Codes](/cli/reference/exit-codes) for the exact value and what to do. Check your balance before a large batch run: ```bash theme={null} testsprite usage ``` Opt out of auto-heal for a rerun to avoid the heal charge when healing isn't needed: ```bash theme={null} testsprite test rerun test_3a9f21c7 --wait --no-auto-heal ``` ## Scopes **Scopes** are permissions that an API key carries. The CLI checks scopes before sending write or run requests, and when access is denied it prints which scope was required and which scopes your key holds. Read commands need read scopes, writes need `write:tests`, and runs need `run:tests`; the purely local `agent install` command needs none, and `setup` works with any valid key. See [Authentication → Scopes](/cli/core/authentication#scopes) for the full table of which scope gates which command. ## Failure Bundle A **failure bundle** is one self-consistent, run-scoped package of everything needed to understand and fix a test failure — assembled by the backend and downloaded by the CLI in a single command. A bundle contains: * The **failing step** and its immediate neighbors (±1 step) * **DOM snapshots** at the point of failure — HTML text your coding agent can read without a vision model, each with a short text description of what the step shows * The **test source** (plan or code) at run time * A **root-cause hypothesis** — the backend's best guess at what broke * A **recommended fix target** — which part of the code to look at * For **backend tests**: the run's captured stdout (`apiOutput`) and Python traceback (`trace`) Every item in a bundle shares one `snapshotId`, so the agent is always reasoning over a single consistent moment in the run. The CLI refuses to stitch data from two different runs. Download the bundle for the latest failing run of a test: ```bash theme={null} testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure ``` `test failure get` follows the latest failure, while `test artifact get ` pins the bundle to one specific run — see [Reading Results](/cli/core/reading-results#pinning-to-a-specific-run) for when to use each. Get a one-screen triage card without downloading the full bundle: ```bash theme={null} testsprite test failure summary test_3a9f21c7 ``` Use `--failed-only` on either command to keep only the failing step and its ±1 neighbors, trimming the bundle to what the agent needs most. ## Where to Go Next The verification loop, idempotency, and where the CLI fits in the bigger picture Create your first test, run it, and read the result end to end Statuses, failure bundles, run history, and artifact downloads Every command, flag, and example in one place # The Agent Loop Source: https://docs.testsprite.com/cli/concepts/the-agent-loop Why the CLI is a loop, not a one-shot -- how your coding agent creates, runs, reads, fixes, and reruns, and how coverage compounds with every pass. The CLI is built around one idea: **verification should be a loop your coding agent runs continuously, not a gate you hit at the end.** Every time the agent changes code, it verifies the behavior it just touched and banks the result. This page explains the mechanics behind that loop. ## The loop ```mermaid theme={null} %%{init: {'flowchart': {'nodeSpacing': 60, 'rankSpacing': 70}, 'themeVariables': {'fontSize': '22px'}}}%% flowchart TD A["🤖 Your coding agent"] D{"behavior already
covered by the suite?"} B["testsprite test create
new behavior → new test"] R["testsprite test rerun
replay the existing tests"] C{{"☁️ TestSprite runs against
real browsers & APIs"}} F["testsprite test failure get
ONE self-consistent bundle"] S[("📚 Durable suite
grows with every pass")] A -->|"writes / changes code"| D D -->|"no — new behavior"| B D -->|"yes"| R B --> C R --> C C -->|"pass ✅"| S C -->|"fail ❌"| F F -->|"agent reads the bundle
& fixes the code"| A S -.->|"defines what's covered"| D ``` ## The four moves The durable suite *is* the answer. If the agent just wrote something new, it isn't covered yet. If it touched existing behavior, the suite already has a test for it. Describe the behavior — as a plan file for frontend tests (`planSteps[]`), or as code (typically Python) for backend tests — then create and run it: ```bash theme={null} testsprite test create \ --project proj_8f0f6 --type frontend \ --plan-from ./checkout-flow.plan.json \ --run --wait --output json ``` `--run --wait` chains create → trigger → poll into one blocking command. Exit 0 means the new test passed and is banked. Replay the existing suite so nothing that used to work breaks silently: ```bash theme={null} testsprite test rerun --all --project proj_8f0f6 --wait ``` Frontend reruns replay the saved script verbatim, billed as a rerun (same price as a fresh run) — [auto-heal](/cli/core/rerun-and-auto-heal) adds a small additional charge only if it engages. On exit 1, pull the failure bundle — one self-consistent package the agent can act on directly: ```bash theme={null} testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure # agent reads the bundle, edits the code… testsprite test rerun test_3a9f21c7 --wait ``` One bundle in, a code fix out, a replay to confirm. The confirmed pass is banked, and the next iteration reruns rather than recreates. **Every pass is banked, not thrown away.** The rerun path is fast and billed the same as a fresh run — use it aggressively. Your agent should rerun the relevant suite after every significant change, not just the test it created last. ## Why this design works An agent reasons over whatever context you hand it. If that context mixes a failing step from one run with source code from a *different* run, the agent will confidently "fix" the wrong thing. `testsprite test failure get` (and `test artifact get`) return a bundle where **every artifact shares one `snapshotId`** — the failing step, its neighbors, the DOM snapshots rendered as text, the test source, and the root-cause hypothesis all describe the same moment. The CLI **refuses** to stitch data across runs or code versions. That's what makes the output safe to feed straight into an agent — no dashboard scraping, no manual screenshot-pasting. Every passing test joins a durable suite — a lasting record of every requirement the agent has ever gotten right, far bigger than any context window. As the project grows, the suite grows with it, and the "already covered?" question gets answered by real, replayable tests rather than the agent's memory. A regression is caught the next time the suite runs, not when a user reports it. You describe intent; the cloud does the work; you read structured results. Your agent never has to know *how* the test was driven — only what a real user experienced. Tests run against your **live product**, not mocks. A frontend test opens a real browser, navigates your app exactly as a user would, and asserts against real behavior. A backend test executes your test code (typically Python) against real API endpoints. This has two consequences: * **No environment setup on your side.** You don't install a browser engine, configure proxies, or manage versions. The cloud handles it. * **Results reflect production reality.** If a test fails, something in the real app is wrong — not a test-harness artifact. The CLI does not support `localhost` targets. Testing a localhost app requires the MCP Server, which manages the tunnel for you. See [MCP Server](/mcp/getting-started/introduction). `--output json` plus stable exit codes form a contract the loop depends on: every command emits the same JSON shape and the same exit codes across releases, so your agent can branch on results without defensive parsing or dashboard scraping. That stability is what makes the loop safe to run unattended. See [Output & Scripting](/cli/reference/output-and-scripting#the-json-contract) for the JSON shape, `--dry-run`, jq, and branching patterns. Write commands — `project create`, `test create`, `test run`, `test rerun` — all carry an **idempotency key**. The backend deduplicates on this key (time-bounded), so retrying a failed network request never creates a duplicate project, test, or run. The CLI generates a random key per invocation by default. Pin your own key to make a command repeatable with guaranteed idempotency: ```bash theme={null} testsprite test create \ --project proj_8f0f6 --type backend \ --name "create order" --code-file ./tests/create_order.py \ --idempotency-key my-agent-step-42 ``` When you replace backend code, a `codeVersion` token guards against silent overwrites — see [Editing & Deleting Tests](/cli/core/editing-tests#editing-a-test). The CLI gives you two ways to reach failure artifacts: `test failure get` follows the *latest* failing run (which can shift if a Portal or scheduled run fires mid-loop), while `test artifact get` is pinned to a specific `runId` and never moves. Which one you pick matters whenever multiple runs might overlap. In agent loops and CI pipelines, always capture the `runId` from `--output json` after triggering a run, then use `test artifact get ` to download artifacts. This prevents the agent from reasoning over a mismatched bundle if another run lands concurrently. See [Reading Results](/cli/core/reading-results#pinning-to-a-specific-run) for the full comparison. ## Where the CLI fits The CLI is one of three surfaces over the same backend and data. See how the Web Portal, MCP Server, and CLI compare. Schedule creation, billing management, crawl/site discovery, and per-step regeneration stay in the Web Portal. The CLI surface is focused on the test lifecycle: create, run, read, fix, rerun. ## Where to Go Next Projects, tests, runs, statuses, credits, scopes, and failure bundles defined Walk through your first test end to end in about 10 minutes Triggering runs, waiting for verdicts, and handling every exit code Let your coding agent drive the loop on its own # Coding Agent Integration Source: https://docs.testsprite.com/cli/core/agent-integration Let your coding agent drive the TestSprite verification loop on its own — create, run, read failures, fix, and rerun without a human in the middle. ## What the skills do `testsprite agent install` writes two **skill files** into your repository by default: * **`testsprite-verify`** — the verification-loop skill. Once in place, your coding agent discovers that this project is TestSprite-tested and knows exactly how to drive the full loop on its own: describe a behavior as a plan file and create a test; trigger a real cloud run and wait for the verdict; on failure, pull one self-consistent bundle — failing step, DOM snapshots rendered as text, root-cause hypothesis, and recommended fix target; edit the code and replay it. * **`testsprite-onboard`** — the onboarding skill. Guides the agent through initial project setup (running `testsprite setup`, creating the first project and test) the first time TestSprite is used in a repo. Both skills are **pure-local**: `agent install` reads and writes only to your filesystem. It makes no network requests and requires no credentials. **We strongly recommend installing the skills.** It's the fastest way to get consistent results — they teach your agent exactly when to verify, how to read the failure bundle, and how to loop until the test is green, so you don't have to spell it out each time. Run `testsprite setup` (or `testsprite agent install`) once per project. ## Install the skills ```bash theme={null} testsprite agent install --target claude ``` Run this from your project root. The skill files (both `testsprite-verify` and `testsprite-onboard` by default) are written immediately. **Flags:** | Flag | Description | | :--------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--target ` | Agent target: `claude`, `cursor`, `cline`, `antigravity`, `kiro`, `windsurf`, `copilot`, `codex`. Comma-separated or repeated. Prompts if omitted in a terminal. | | `--skill ` | Skill to install: `testsprite-verify`, `testsprite-onboard`. Comma-separated or repeated. Default: both. | | `--dir ` | Project root to write the skill(s) into (default: current directory). | | `--force` | Overwrite an existing skill file. A `.bak` backup is kept. For `codex`, replaces only the managed section — your other `AGENTS.md` content is preserved. | If `--target` is omitted in an interactive terminal, the CLI prompts for one (default `claude`). Outside a terminal — CI, a script, a piped invocation — it skips the prompt and defaults straight to `claude`, printing an `[info]` line to stderr so the choice is visible in logs. To install for multiple agents at once: ```bash theme={null} testsprite agent install --target claude --target cursor ``` Or comma-separated: ```bash theme={null} testsprite agent install --target claude,cursor ``` If the destination file already exists and differs from the current skill, the install is blocked until you pass `--force`. The original is always backed up as `.bak`. ## Supported agents | Target | Status | Mode | Landing path | | :------------ | :---------------------- | :-------------- | :--------------------------------------------- | | `claude` | GA | own-file | `.claude/skills/{skill}/SKILL.md` | | `antigravity` | experimental | own-file | `.agents/skills/{skill}/SKILL.md` | | `cursor` | experimental | own-file | `.cursor/rules/{skill}.mdc` | | `cline` | experimental | own-file | `.clinerules/{skill}.md` | | `kiro` | experimental | own-file | `.kiro/skills/{skill}/SKILL.md` | | `windsurf` | experimental | own-file | `.windsurf/rules/{skill}.md` | | `copilot` | experimental | own-file | `.github/instructions/{skill}.instructions.md` | | `codex` | experimental | managed-section | `AGENTS.md` | `{skill}` is `testsprite-verify` or `testsprite-onboard`. Own-file targets write one file per installed skill; only the requested skill(s) land, so `--skill testsprite-verify` writes just that one file. For `codex`, every installed skill is merged into one managed section inside `AGENTS.md`. Your existing content in that file is not modified — only the TestSprite section is added or updated. ## Listing targets To see all supported targets, their current status, and where each skill lands: ```bash theme={null} testsprite agent list ``` ```text theme={null} TARGET SKILL STATUS MODE PATH claude testsprite-verify GA own-file .claude/skills/testsprite-verify/SKILL.md claude testsprite-onboard GA own-file .claude/skills/testsprite-onboard/SKILL.md antigravity testsprite-verify experimental own-file .agents/skills/testsprite-verify/SKILL.md antigravity testsprite-onboard experimental own-file .agents/skills/testsprite-onboard/SKILL.md cursor testsprite-verify experimental own-file .cursor/rules/testsprite-verify.mdc cursor testsprite-onboard experimental own-file .cursor/rules/testsprite-onboard.mdc cline testsprite-verify experimental own-file .clinerules/testsprite-verify.md cline testsprite-onboard experimental own-file .clinerules/testsprite-onboard.md kiro testsprite-verify experimental own-file .kiro/skills/testsprite-verify/SKILL.md kiro testsprite-onboard experimental own-file .kiro/skills/testsprite-onboard/SKILL.md windsurf testsprite-verify experimental own-file .windsurf/rules/testsprite-verify.md windsurf testsprite-onboard experimental own-file .windsurf/rules/testsprite-onboard.md copilot testsprite-verify experimental own-file .github/instructions/testsprite-verify.instructions.md copilot testsprite-onboard experimental own-file .github/instructions/testsprite-onboard.instructions.md codex testsprite-verify experimental managed-section AGENTS.md codex testsprite-onboard experimental managed-section AGENTS.md ``` ## Checking skill health Once skills are installed, confirm they're still in sync with the CLI version that installed them — useful right after upgrading the CLI, or in CI to catch a hand-edited skill file: ```bash theme={null} testsprite agent status ``` Each installed skill file is classified into one state: | State | Meaning | | :--------- | :-------------------------------------------------------------------------- | | `ok` | Matches exactly what this CLI version would install | | `stale` | Installed by an older CLI version whose canonical content has since changed | | `modified` | Edited by hand after install | | `unmarked` | Present, but predates the provenance marker this check relies on | | `absent` | Not installed for this target/skill | | `corrupt` | `codex` only — the managed `AGENTS.md` section's sentinels are malformed | Pass `--dir ` to inspect a project other than the current directory. `agent status` exits `1` when anything needs attention (any state other than `ok` or `absent`), so it's CI-gateable: ```bash theme={null} testsprite agent status --dir ./apps/web && echo "skills are in sync" ``` ## One-shot onboarding `testsprite setup` configures your API key **and** installs the skills in a single step — the recommended path when you're setting up a project for the first time. ```bash theme={null} testsprite setup ``` `setup` chains credential configuration → identity verification → agent skill install, and prints a unified summary. The full onboarding walkthrough, including the non-interactive (`--from-env --yes`) form for CI bootstrap. ## How the agent uses it Once the skill files are in place, a coding agent like Claude Code picks them up automatically when it enters your project. It knows to: 1. **Create** a test from a plan you or the agent authors — describing the behavior in intent terms, not driver code. 2. **Run** it against the live app and wait for a pass/fail verdict. 3. On failure, **pull the failure bundle** — one consistent artifact with everything needed to diagnose and fix. 4. **Fix** the code, then **rerun** as a verbatim replay. The command sequence the agent follows: ```bash theme={null} # Create a test from a plan file and immediately run it testsprite test create \ --project proj_8f0f6 --type frontend \ --plan-from ./checkout-flow.plan.json \ --run --wait --output json # On failure: pull the self-consistent failure bundle testsprite test failure get test_3a9f21c7 \ --out ./.testsprite/failure # After fixing the code: replay verbatim (billed as a rerun, same as a fresh run) testsprite test rerun test_3a9f21c7 --wait ``` Exit 0 means the test passed and is banked into the durable suite. The agent doesn't need to know how the test was driven — only what a real user experienced. Every passing rerun compounds your coverage. The agent builds a lasting record of every requirement it has verified — far bigger than any context window. ## Where to Go Next Walk through the full create → run → fix loop end-to-end Gate your pipeline on TestSprite results All flags and modes for triggering and waiting on runs Full flag reference for every command # Authentication Source: https://docs.testsprite.com/cli/core/authentication Configure an API key, check your active identity, and manage named profiles for multiple accounts or environments. ## Signing in Before you can run any command that talks to the TestSprite API, you need to store a valid API key. The fastest path is `testsprite setup` — it prompts for the key, verifies it against the server, installs the agent skill into your repo, and prints a unified summary, all in one step: ```bash theme={null} testsprite setup ``` The full onboarding walkthrough If you want credentials only — no skill install — add `--no-agent`: ```bash theme={null} testsprite setup --no-agent ``` The CLI calls `GET /me` with the key you provide before writing anything to disk. A rejected or malformed key never overwrites a working profile. You are only ever prompted for the API key. ```text theme={null} TestSprite API key: ******** TestSprite initialized. profile: default env: production email: you@example.com scopes: read:me, read:projects, read:tests, write:tests, run:tests agent: skipped (--no-agent) ``` For non-interactive environments (CI, Dockerfiles, scripted setup), pass `--from-env` instead of typing at a prompt: ```bash theme={null} TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yes --no-agent ``` **Important:** The CLI never accepts an API key as an inline positional argument or flag — that would expose it in shell history and process listings. Use the interactive prompt, `--from-env`, or the `TESTSPRITE_API_KEY` environment variable. ## Checking who you are Run `testsprite auth status` to confirm which key and profile are active: ```bash theme={null} testsprite auth status ``` Sample text output: ```text theme={null} userId usr_1478d468 name Your Name email you@example.com keyId key_a1b2c3d4 env production scopes read:me, read:projects, read:tests, write:tests, run:tests ``` If your key is missing `write:tests` or `run:tests`, the output appends a `note:` line listing the gap. For machine-readable output, add `--output json`. ## Signing out To remove credentials for the active profile: ```bash theme={null} testsprite auth remove ``` The credentials entry for the selected profile is deleted from `~/.testsprite/credentials`. Other profiles are not touched. ## Where credentials live The CLI stores credentials in `~/.testsprite/credentials`, an INI-style file. The directory is created with mode `0700` and the file with mode `0600`. All writes are atomic. ```ini theme={null} [default] api_key = sk-... [ci] api_key = sk-... ``` Each section is a named **profile**. The section name matches the profile you activated when you ran `setup`. ## Profiles Profiles let you manage multiple accounts or API keys (for example, an interactive key and a CI key) without re-entering credentials each time. To configure a named profile, pass `--profile` before the subcommand: ```bash theme={null} testsprite --profile ci setup --no-agent ``` To use a profile for any subsequent command, pass `--profile` as a global flag: ```bash theme={null} testsprite --profile ci project list ``` Or set the environment variable so every command in the session picks it up automatically: ```bash theme={null} export TESTSPRITE_PROFILE=ci testsprite project list ``` The active profile determines which `api_key` is read from `~/.testsprite/credentials`. If the named section does not exist, the CLI exits with a validation error (exit 5). ## Environment variables Environment variables override the credentials file. This is the recommended pattern for CI and container environments. | Variable | Purpose | Precedence | | :------------------- | :----------------------------------------- | :----------------------------- | | `TESTSPRITE_API_KEY` | API key to use for all requests | Wins over the credentials file | | `TESTSPRITE_PROFILE` | Active profile name (default: `"default"`) | Overridden by `--profile` | Full resolution order: * **Profile:** `--profile` flag > `TESTSPRITE_PROFILE` env > `"default"` * **API key:** `TESTSPRITE_API_KEY` env > credentials file value for the active profile For the complete precedence table including the request-timeout setting, see [Configuration](/cli/reference/configuration). ## Scopes API keys carry a list of **scopes** that gate which operations the CLI can perform. All new keys and grandfathered keys default to the full working set. | Scope | Gates | | :-------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `read:me` | `auth status`, `usage` | | `read:projects` | `project list`, `project get` | | `read:tests` | `test list`, `test get`, `test steps`, `test result`, `test code get`, `test failure get`, `test failure summary`, `test artifact get` | | `write:tests` | `test create`, `test create-batch`, `test update`, `test delete`, `test delete-batch`, `test code put`, `test plan put`, `project create`, `project update` | | `run:tests` | `test run`, `test wait`, `test rerun` | `testsprite agent install` is a pure-local operation — it only writes files to your project directory and never calls the API, so no scope is required. `testsprite setup` *does* call the API to verify your key (`GET /me`) and fetch your identity, but it works with any valid key and needs no write or run scope. If a command fails with a scope error, the CLI prints the required and granted scopes in both text and JSON modes. You can generate a new key with the needed scopes in the dashboard at Settings → API Keys → Create new key. The Web Portal key management walkthrough ## Where to Go Next Create and manage the projects your tests live in Author frontend plans and backend code files Full flag and environment variable precedence reference Non-interactive auth patterns for pipelines # Cancelling a Run Source: https://docs.testsprite.com/cli/core/cancelling-runs Ctrl-C detaches, it never cancels — testsprite test cancel is the real stop button. What each one does to the run, your credits, and your exit codes. Two different things can end your involvement with a run in flight, and they are deliberately not the same: | Action | What stops | What keeps going | | :-------------------------------- | :-------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Ctrl-C** during `--wait` | Only your local CLI (it detaches) | The run — it keeps executing **and billing** on the server | | `testsprite test cancel ` | The run, server-side | Only work already started in the cloud, which finishes in the background with its result discarded — the run stays cancelled and the test is immediately free to run again | ## Ctrl-C is a detach, not a cancel Pressing Ctrl-C while `test run --wait`, `test wait`, `test rerun --wait`, or `test flaky` is polling stops the *watching*, never the *run*. The CLI tells you exactly that on the way out: ```text theme={null} Interrupted (SIGINT). Run run_5c1d9a2b is still executing on the server and will keep running (and billing) until it finishes. Re-attach with: testsprite test wait run_5c1d9a2b Cancel with: testsprite test cancel run_5c1d9a2b ``` Just like a `--timeout` expiry, the CLI prints a partial object (`{ "runId": "...", "status": "running" }`) to stdout before exiting, so a script that gets interrupted still captures the `runId`. The exit code is the conventional `128 + signal`: `130` for SIGINT (Ctrl-C), `143` for SIGTERM, `129` for SIGHUP. In `--output json` mode, stderr carries a machine-readable envelope instead of prose: ```json theme={null} { "error": { "code": "INTERRUPTED", "message": "Interrupted by SIGINT.", "nextAction": "The server-side run (if any) keeps executing and billing. Re-attach with: testsprite test wait , or stop it with: testsprite test cancel (runId is in the partial JSON on stdout).", "requestId": "local", "details": { "signal": "SIGINT" } } } ``` A second Ctrl-C while the first is being handled exits immediately, no cleanup — the escape hatch when even the goodbye message is too slow. Interrupting a **batch** (`test run --all --wait`, multi-id `test wait`/`test rerun`) prints the same partial listing every run that was already dispatched, so nothing already billed goes unaccounted for. ## Actually stopping a run ```bash theme={null} testsprite test cancel run_5c1d9a2b ``` The run flips to cancelled and the CLI prints the final run card: ```text theme={null} runId run_5c1d9a2b testId test_3a9f21c7 status cancelled createdAt 2026-07-10T02:11:04Z startedAt 2026-07-10T02:11:09Z failureKind user_cancel ``` What cancel does — and deliberately does not do: * **The test is immediately re-runnable.** Cancelling frees the test's run slot, so a follow-up `test run` won't hit the "run already in flight" conflict. * **The test's last verdict is untouched.** A test that was passed before the cancelled run stays passed — cancelling never overwrites history with a fake failure. * **No refund.** Credits charged when the run was triggered stay charged. Cancelling limits *future* cost (a pending auto-heal pass that hasn't engaged yet won't, so its fee is never charged) but never claws back a charge already made. * **The engine may finish in the background.** Work already started in the cloud can run to completion, but its result is discarded — it will never overwrite the cancelled status or the test's verdict. * **Cancelled runs stay in history.** `testsprite test result --history` lists them like any other run. ## Idempotent by design Cancelling the same run twice is a success, not an error — the second call exits `0` with an advisory: ```text theme={null} [advisory] run run_5c1d9a2b was already cancelled ``` That makes cancel safe to retry blindly from scripts and cleanup traps. Only two things are refused: | Situation | Exit | Why | | :-------------------------------------------------------------------------------- | :--- | :--------------------------------------------------------- | | Run already finished (passed / failed / blocked) | 6 | There is nothing left to stop — the verdict already exists | | Run ID unknown (or belongs to another tenant) | 4 | Check the ID | ## Cancelling several runs at once `test cancel` is variadic — after interrupting a batch, paste every ID from the hint: ```bash theme={null} testsprite test cancel run_5c1d9a2b run_7f2e1c04 run_9a01b3e5 ``` Multi-id output is a summary instead of a run card: ```json theme={null} { "cancelled": ["run_5c1d9a2b", "run_7f2e1c04"], "alreadyCancelled": [], "conflicts": [{ "runId": "run_9a01b3e5", "status": "passed" }], "notFound": [], "errors": [] } ``` The exit code reports the *worst* outcome: any `notFound` → `4` (a wrong ID is a caller bug worth failing loudly on), else any transport/auth `errors` → `1`, else any `conflicts` → `6`, else `0`. Runs that were fresh-cancelled or already cancelled both count as success. ## CI cleanup pattern Cancel's idempotency makes it a natural `trap` target — detach honestly on interrupt, then stop the spend: ```bash theme={null} RUN_ID=$(testsprite test run test_3a9f21c7 --output json | jq -r '.runId') trap 'testsprite test cancel "$RUN_ID"' INT TERM testsprite test wait "$RUN_ID" ``` ## Where to Go Next Triggering runs, waiting for verdicts, and resuming The full exit-code and signal contract Run history, steps, and failure bundles Wire the CLI into GitHub Actions or any pipeline # Creating Tests Source: https://docs.testsprite.com/cli/core/creating-tests Author frontend test plans and backend test code, then create individual tests or batches from the CLI. ## Two kinds of tests TestSprite has two test types, and the authoring model differs between them: | | **Frontend** | **Backend** | | :-------------------- | :------------------------------------------------------------------------------------ | :------------------------------------------------------------------------------------- | | What it is | An ordered list of steps — interactions a real browser performs against your live app | Code that calls your API and asserts on the responses — typically Python with `pytest` | | You author | A plan file (JSON) | The test code locally | | Pass it with | `--plan-from` | `--code-file` | | TestSprite runs it as | A real browser session in the cloud | Code in an isolated cloud sandbox | TestSprite stores and executes all test code as Python: frontend tests run as async Playwright scripts, backend tests as `requests` + pytest-style assertions. Accordingly, `test code put --language` accepts only `python`. ## Create a frontend test from a plan ```bash theme={null} testsprite test create \ --plan-from ./checkout-flow.plan.json ``` The plan file is a JSON document that holds the full test definition — `projectId`, `type`, `name`, and the `planSteps` array. Because everything is in the file, `--project`, `--type`, `--name`, `--description`, and `--priority` are ignored when `--plan-from` is set; use the fields inside the JSON instead. Plan files must be ≤ 256 KB. Example plan file structure: ```json theme={null} { "projectId": "proj_8f0f6", "type": "frontend", "name": "Guest checkout — credit card", "planSteps": [ { "type": "action", "description": "Navigate to https://app.example.com/cart" }, { "type": "action", "description": "Click the checkout button" }, { "type": "action", "description": "Fill the card number field with a test card number" }, { "type": "action", "description": "Click the Pay button" }, { "type": "assertion", "description": "The order confirmation message is visible" } ] } ``` On success, the CLI prints the new `testId`, `codeVersion`, `createdAt`, and — when the backend supplies it — a `dashboardUrl` deep-linking to the Portal. ## Create a backend test from code ```bash theme={null} testsprite test create \ --project proj_8f0f6 \ --type backend \ --name "Create order" \ --code-file ./tests/create_order.py ``` Code files must be ≤ 350 KB. In code-file mode, `--project`, `--type`, and `--name` are all required. | Flag | Description | | :---------------------------- | :-------------------------------------------------------------------------------------------- | | `--project ` | **Required.** Project this test belongs to | | `--type ` | **Required.** Test type | | `--name ` | **Required.** Display name (≤ 200 characters) | | `--code-file ` | **Required.** File containing the test code (≤ 350 KB). Mutually exclusive with `--plan-from` | | `--description ` | Optional description (≤ 2000 characters) | | `--priority ` | Optional priority level | ## Backend dependency authoring Backend tests can declare the variables they produce and consume. TestSprite uses these declarations to determine run order — producers execute before consumers, and teardown tests run last — both on `test run --all` and on rerun. | Flag | Description | | :----------------- | :--------------------------------------------------------------------------------------------- | | `--produces ` | Variable this test captures (repeatable). Example: `--produces orderId` | | `--needs ` | Variable this test requires from an upstream producer (repeatable). Example: `--needs orderId` | | `--category ` | Use `teardown` or `cleanup` to mark a final-wave cleanup test | These flags are backend-only and are ignored for frontend tests. ```bash theme={null} # Producer: creates an order, exposes orderId downstream testsprite test create \ --project proj_8f0f6 --type backend \ --name "Create order" \ --code-file ./tests/create_order.py \ --produces orderId # Consumer: needs orderId from the producer above testsprite test create \ --project proj_8f0f6 --type backend \ --name "Fetch order details" \ --code-file ./tests/fetch_order.py \ --needs orderId # Teardown: runs after all others, cleans up state testsprite test create \ --project proj_8f0f6 --type backend \ --name "Delete test orders" \ --code-file ./tests/cleanup.py \ --category teardown ``` ## Create and run in one command Pass `--run` to trigger a cloud run immediately after the test is created. Add `--wait` to block until the run reaches a terminal status. Add `--output json` to get a machine-readable result your agent can parse. ```bash theme={null} testsprite test create \ --project proj_8f0f6 \ --type frontend \ --plan-from ./checkout-flow.plan.json \ --run --wait \ --output json ``` Exit 0 means the run passed. Any non-zero exit means the run failed, was blocked, or timed out — your script can branch directly on `$?`. See [Running Tests](/cli/core/running-tests) for the full run surface, including `--timeout`, `--target-url`, and how to resume a timed-out run with `test wait `. ## Creating many tests at once Use `test create-batch` to create up to 50 frontend tests in a single call. This command is frontend-only. ```bash theme={null} # From a JSONL file — one plan-from spec per line testsprite test create-batch --plans ./flows.jsonl # From a directory of *.json plan files, sorted by filename testsprite test create-batch --plan-from-dir ./plans/ ``` | Flag | Description | | :-------------------------- | :------------------------------------------------------------------------------------------------------ | | `--plans ` | JSONL file, one plan spec per line (≤ 50 specs, ≤ 5 MB). Mutually exclusive with `--plan-from-dir` | | `--plan-from-dir ` | Directory of `*.json` plan files, one spec each (≤ 50 files, ≤ 5 MB total; processed in filename order) | | `--run` | Trigger a run for each created test | | `--wait` | With `--run`, block until all runs reach a terminal status | | `--timeout ` | With `--run --wait`, max seconds to wait (default 600) | | `--target-url ` | With `--run`, override the project default target URL for all triggered runs | | `--max-concurrency ` | With `--run`, max in-flight run triggers (1–100, default 50) | | `--idempotency-key ` | Pin for safe retries | The server caps run triggers at 60 per minute per key. The CLI throttles to 50 per minute and auto-retries rate-limited requests — you do not need to handle this yourself. Editing a test's metadata, replacing its plan or code, and deleting tests all live in [Editing & Deleting Tests](/cli/core/editing-tests). ## Where to Go Next Update metadata, replace a plan or code, and delete tests Trigger runs, poll for verdicts, and handle timeouts Fetch failure bundles, steps, and run history Full flag listing for every command # Editing & Deleting Tests Source: https://docs.testsprite.com/cli/core/editing-tests Update a test's metadata, replace its plan steps or code safely, and delete tests one at a time or in bulk. Once a test exists, you can change its metadata, replace its plan or code, or remove it — all from the CLI. Author new tests from the CLI. ## Editing a test **Metadata** (name, description, priority) — and, for backend tests, the **dependency declarations** (`--produces` / `--needs` / `--category`, repeatable; echoed back by `test get`) — use `test update`: ```bash theme={null} testsprite test update test_3a9f21c7 --name "Guest checkout v2" --priority p0 ``` **Frontend plan steps** — use `test plan put`: ```bash theme={null} testsprite test plan put test_3a9f21c7 --steps ./updated-steps.json ``` The `--steps` file must contain a JSON object with a `planSteps` array (≤ 200 steps, ≤ 256 KB). Pass `--expected-step-count ` as an optional concurrency check — the server returns 412 if the current step count differs, preventing a silent overwrite. **Backend test code** — use `test code put`: ```bash theme={null} # Safe update: supply the codeVersion you read from the last create/update testsprite test code put test_3a9f21c7 \ --code-file ./tests/create_order_v2.py \ --expected-version v3 # Force overwrite regardless of current version (audit-logged) testsprite test code put test_3a9f21c7 \ --code-file ./tests/create_order_v2.py \ --force ``` `--expected-version` checks the `codeVersion` (e.g. `v3`) you received from the last create or update. If the test has a newer version, the CLI exits with a conflict (exit 6) — re-fetch the current code and retry. `--force` overwrites regardless of the current version. The two flags are mutually exclusive. **codeVersion** is an opaque version token that changes on every code change. It prevents two agents (or a human and an agent) from silently overwriting each other's edits. More on safe concurrent edits. ## Deleting tests Delete a single test — `--confirm` is required: ```bash theme={null} testsprite test delete test_3a9f21c7 --confirm ``` Delete multiple tests in one call: ```bash theme={null} # By explicit IDs testsprite test delete-batch test_3a9f21c7 test_b1e04f2a --confirm # All tests in a project testsprite test delete-batch --all --project proj_8f0f6 --confirm # All failed tests in a project testsprite test delete-batch --all --project proj_8f0f6 --status failed --confirm ``` The CLI prints a summary line: `Deleted N, Skipped M, Failed K`. A 404 response counts as skipped rather than an error, so it is safe to pass IDs that may have already been deleted. Exit codes: 0 if all targeted tests were deleted or skipped; 1 if any deletion failed; 5 for validation errors. ## Where to Go Next Author frontend plans and backend code, single or in batches Trigger runs, poll for verdicts, and handle timeouts Fetch failure bundles, steps, and run history Full flag listing for every command # Projects Source: https://docs.testsprite.com/cli/core/projects List, inspect, create, and update the typed (frontend or backend) projects that hold your tests. A **project** is the top-level container in TestSprite. It carries a name, a type (frontend or backend), and — for frontend projects — a target URL that TestSprite opens when it runs your tests. Every test belongs to exactly one project, identified by its `projectId`. Projects created or updated through the CLI are immediately visible in the Web Portal, and vice versa. All surfaces share the same data. ## Listing projects ```bash theme={null} testsprite project list ``` Text output is a table with five columns: ```text theme={null} ID NAME TYPE FROM CREATED proj_8f0f6 Checkout App frontend cli 2026-05-10 proj_a3c12 Orders API backend portal 2026-04-22 ``` | Flag | Description | | :------------------------- | :------------------------------------------------------------ | | `--page-size ` | Number of results per page (1–100, default 25) | | `--starting-token ` | Opaque cursor from a previous response to fetch the next page | | `--max-items ` | Stop after this many total items across auto-paged requests | When there are more results, the CLI appends a `nextToken` line to the text output. Pass that value as `--starting-token` on the next call to continue. ```bash theme={null} testsprite project list --page-size 10 --starting-token eyJsYXN... ``` ## Getting a project ```bash theme={null} testsprite project get ``` Text output: ```text theme={null} id proj_8f0f6 name Checkout App type frontend createdFrom cli createdAt 2026-05-10T14:32:00Z updatedAt 2026-05-11T09:15:00Z ``` ## Creating a project ```bash theme={null} testsprite project create --type frontend --name "Checkout App" --url https://app.example.com ``` | Flag | Description | | :--------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------- | | `--type ` | **Required.** Project type | | `--name ` | **Required.** Display name for the project | | `--url ` | **Required for frontend.** Target URL — must be a public `http://` or `https://` address. Localhost and private IP ranges are not accepted | | `--username ` | Optional basic-auth username for the target URL | | `--password ` | Optional basic-auth password (prefer `--password-file` in scripts) | | `--password-file ` | Read the password from a file — safer than inline for non-interactive use | | `--instruction ` | Optional hint passed to the frontend plan-generation step | | `--idempotency-key ` | Defaults to a UUIDv4 per invocation; pin a stable value for safe retries | **Frontend project example:** ```bash theme={null} testsprite project create \ --type frontend \ --name "Checkout App" \ --url https://app.example.com \ --instruction "Focus on the cart and payment pages" ``` **Backend project example:** ```bash theme={null} testsprite project create --type backend --name "Orders API" ``` `--url` must be a public `http(s)` address — `localhost` and private IPs are rejected before any network call. To test a local app, use the [MCP Server](/mcp/getting-started/introduction); see [Common Issues](/cli/troubleshooting/common-issues) if a URL is rejected. ## Updating a project ```bash theme={null} testsprite project update --name "Checkout App v2" ``` You must pass at least one mutable flag — an update with no changes exits with a validation error (exit 5). | Flag | Description | | :-------------------------- | :-------------------------------------------------------------- | | `--name ` | New display name | | `--url ` | New target URL (frontend only; same public-URL rules as create) | | `--username ` | Updated basic-auth username | | `--password ` | Updated password (prefer `--password-file`) | | `--password-file ` | Read updated password from a file | | `--instruction ` | Updated plan-generation hint | | `--idempotency-key ` | Pin for safe retries | ## Backend test credentials Backend projects can inject auth into every backend test run, either as a static credential or as a recurring, auto-refreshed token: ```bash theme={null} # Static credential (free tier) — Bearer token, API key, or Basic auth testsprite project credential --type "Bearer token" --credential-file ./token.txt # Recurring-token auto-refresh login (Pro) — fetches a fresh token on every run testsprite project auto-auth --method refresh_token --inject bearer \ --token-endpoint https://auth.example.com/oauth/token \ --client-id my-client-id --client-secret-file ./client-secret.txt \ --refresh-token-file ./refresh-token.txt ``` Full flag reference for `project credential` and `project auto-auth` ## Where to Go Next Author a frontend plan or backend code file inside a project Project, test, run — definitions and relationships Trigger runs and poll for verdicts Full flag listing for every command # Reading Results Source: https://docs.testsprite.com/cli/core/reading-results Inspect individual tests, step-by-step execution logs, run history, and failure bundles from the CLI. ## Inspecting a test Get the current state of a single test by ID: ```bash theme={null} testsprite test get test_3a9f21c7 ``` The response shows the test's status, type, plan step count, when it was created, and which surface last authored it: ```text theme={null} id: test_3a9f21c7 projectId: proj_8f0f6 name: Checkout flow — guest user type: frontend createdFrom: cli status: passed planStepCount: 12 createdAt: 2026-06-10T08:14:22Z updatedAt: 2026-06-15T10:04:38Z ``` List all tests in a project with filters: ```bash theme={null} testsprite test list --project proj_8f0f6 ``` Narrow by type, originating surface, or status: ```bash theme={null} testsprite test list --project proj_8f0f6 --type frontend --status failed,blocked testsprite test list --project proj_8f0f6 --created-from cli ``` | Flag | Values | Description | | :------------------------- | :------------------------- | :--------------------------------------------------------------------------------------------------- | | `--type` | `frontend` \| `backend` | Filter by test type | | `--created-from` | `portal` \| `mcp` \| `cli` | Filter by authoring surface | | `--status` | comma-separated list | Any of: `draft`, `ready`, `queued`, `running`, `passed`, `failed`, `blocked`, `cancelled`, `unknown` | | `--page-size ` | 1–100, default 25 | Items per page | | `--starting-token ` | opaque cursor | Continue from a previous page | ## Step-by-step results `test steps` returns the cumulative execution log for a test — every step from every run, in order: ```bash theme={null} testsprite test steps test_3a9f21c7 ``` Scope the output to a single run with `--run-id`: ```bash theme={null} testsprite test steps test_3a9f21c7 --run-id run_5c1d9a2b ``` Each entry shows the step name, its status (passed or failed), and the run it belongs to. Use `--page-size` and `--max-items` for large histories. Steps logged before run IDs were tracked are excluded when you supply `--run-id`. If you need those older entries, query without the flag. ## The latest result `test result` returns the outcome of the most recent completed run for a test: ```bash theme={null} testsprite test result test_3a9f21c7 ``` Add `--include-analysis` to attach the AI triage block — root-cause hypothesis, recommended fix target, failure kind, and the snapshot ID that ties all artifacts together: ```bash theme={null} testsprite test result test_3a9f21c7 --include-analysis ``` For **backend tests**, the result also surfaces the run's captured stdout (`apiOutput`) and Python traceback (`trace`) — full content under `--output json` (and inside failure bundles as `result.json` / `failure.json`); text mode prints a bounded 20-line tail of each. The analysis fields are especially useful when piping to an agent: ```bash theme={null} testsprite test result test_3a9f21c7 --include-analysis --output json \ | jq '{hypothesis: .analysis.rootCauseHypothesis, fix: .analysis.recommendedFixTarget}' ``` ## Run history Pass `--history` to list prior runs instead of the latest result: ```bash theme={null} testsprite test result test_3a9f21c7 --history ``` Filter by trigger source or time window: ```bash theme={null} testsprite test result test_3a9f21c7 --history --source cli --since 7d testsprite test result test_3a9f21c7 --history --source portal --since 2026-06-10T00:00:00Z ``` Paginate long histories: ```bash theme={null} testsprite test result test_3a9f21c7 --history --page-size 10 --cursor ``` | Flag | Values | Description | | :----------------- | :---------------------------------------------------------- | :------------------------------- | | `--source` | `cli` \| `portal` \| `mcp` \| `schedule` \| `github_action` | Filter by what triggered the run | | `--since` | `24h`, `7d`, or ISO timestamp | Earliest `createdAt` to include | | `--page-size ` | 1–100, default 20 | Items per page | | `--cursor ` | opaque cursor | Continue from a previous page | ## Triaging a failure When a test is failed, start with the one-screen triage card: ```bash theme={null} testsprite test failure summary test_3a9f21c7 ``` This prints the status, failure kind, root-cause hypothesis, and recommended fix target — everything you need to decide what to do next, without downloading anything. To pull the full failure bundle to disk: ```bash theme={null} testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure ``` The bundle is one self-consistent, run-scoped package — failing step, DOM snapshots as text, test source, and root-cause analysis, all anchored to one `snapshotId`. See [Failure Bundle](/cli/concepts/key-terms#failure-bundle) for the full contents. Add `--failed-only` to keep only the failing step and its neighbors (±1), trimming the bundle for faster agent context loading: ```bash theme={null} testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure --failed-only ``` ## Pinning to a specific run `test failure get` always returns the *latest* failing run for a test — that pointer moves if a new run comes in. When multiple runs might be in flight simultaneously, pin to an exact run with `test artifact get`: ```bash theme={null} testsprite test artifact get run_5c1d9a2b --out ./.testsprite/runs/run_5c1d9a2b ``` The artifact bundle for a specific run is immutable — a concurrent Portal or schedule run cannot overwrite it. This is the safe path for agents and CI pipelines where two runs of the same test might overlap. Use `test failure get ` for the interactive triage loop. Use `test artifact get ` in scripts and CI where you need a stable, run-scoped bundle that a concurrent run can't shift under you. The default output directory is `./.testsprite/runs//`. The parent directory must exist before you run the command. ## Reading test code Print the generated test code to stdout: ```bash theme={null} testsprite test code get test_3a9f21c7 ``` Save it to a file instead: ```bash theme={null} testsprite test code get test_3a9f21c7 --out ./tests/checkout_flow.py ``` The response includes the language, framework, code content, and the `codeVersion` token — useful if you plan to update the code with `testsprite test code put` and want protection against overwriting a concurrent edit. ## Where to Go Next Trigger runs, wait for verdicts, and understand exit codes Replay a test or let AI repair UI drift runId, codeVersion, snapshotId, and the rest of the vocabulary Full flag listing for every CLI command # Rerun & Auto-Heal Source: https://docs.testsprite.com/cli/core/rerun-and-auto-heal Replay a saved test, batch-rerun a whole suite, and let AI repair UI drift automatically — all from the CLI. ## What rerun does **Rerun** re-executes a test using what's already saved — no new plan generation, no new code generation. | Test type | What rerun executes | | :------------------------ | :-------------------------------------------------------------------------------------------------------------------------------- | | Frontend tests | Replay the saved Playwright script verbatim against the live app. Billed the same as a fresh run — see [Credits](#credits) below. | | Backend tests | Re-run the full dependency closure: producers first, then the named test, then teardown — in the correct wave order. | This makes rerun the primary iteration loop once a test exists: fix code, rerun, read the verdict, repeat. ## Rerunning a test ```bash theme={null} testsprite test rerun test_3a9f21c7 --wait ``` Without `--wait`, the command returns once the rerun is accepted. With `--wait`, it blocks until passed, failed, blocked, or cancelled — same exit codes as `test run --wait`. Control the polling ceiling: ```bash theme={null} testsprite test rerun test_3a9f21c7 --wait --timeout 300 ``` ## Rerunning many Rerun every test in a project at once: ```bash theme={null} testsprite test rerun --all --project proj_8f0f6 --wait ``` Useful flags for batch rerun: | Flag | Description | | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------- | | `--skip-terminal` | Skip tests that are already in a terminal state (passed, failed, blocked, cancelled) | | `--status ` | Only rerun tests matching these statuses (comma-separated) | | `--filter ` | Case-insensitive name substring filter | | `--max-concurrency ` | Max simultaneous reruns (1–100, default 50) | Example — rerun only the tests that failed, filtered to the checkout group: ```bash theme={null} testsprite test rerun --all --project proj_8f0f6 \ --status failed --filter "checkout" --wait ``` ## Backend dependency closure When you rerun a backend test, the CLI expands the run to include: 1. **Producer tests** — any test that `--produces` a variable the named test `--needs`. 2. **The named test itself.** 3. **Teardown tests** — any test tagged `--category teardown` in the same project. This ensures the test runs in a valid environment, not against stale or missing upstream state. To skip the expansion and rerun just the named test in isolation: ```bash theme={null} testsprite test rerun test_3a9f21c7 --skip-dependencies --wait ``` **Important:** `--skip-dependencies` can cause a backend test to fail if it relies on data created by a producer test. Use it only when you know the required fixtures are already in place. ## Auto-heal **Auto-heal** is on by default for every frontend rerun, on every plan tier, when triggered through the CLI. When your app's UI has shifted since the test was last written — a button was renamed, a form gained an extra field, navigation was restructured — the verbatim script would fail even though the underlying feature still works. Auto-heal detects that drift and repairs the script so the test passes. If the feature itself is broken, the test stays failed. **Auto-heal is on by default for all CLI reruns.** The rerun itself is billed the same as a fresh run regardless of whether healing engages; healing that actually repairs a step consumes an additional small amount of credit on top of that. Opt out with `--no-auto-heal` (rolling out on newer accounts — see the note below). See [Billing & Plans](/web-portal/admin/billing-and-plans) for current rates. The `--no-auto-heal` opt-out is still rolling out to accounts on the newer execution platform — if you rely on strictly disabling auto-heal (for example, in `test flaky`), verify with `testsprite auth status` and a sample rerun. To opt out for a specific rerun: ```bash theme={null} testsprite test rerun test_3a9f21c7 --wait --no-auto-heal ``` Auto-heal is ignored for backend tests — backend test failures are almost always real assertion or fixture issues, not UI drift. **Minor UI refactors that moved or renamed things.** A button relabeled from "Continue" to "Next", a sidebar collapsed into a hamburger menu, a custom dropdown replaced with a design-system component. The feature still works; the saved script just refers to stale selectors or text. Auto-heal re-binds the test to the new UI in context and marks it passed. **A real product bug.** The form rejects valid input, an API returns 500, the redirect lands on the wrong page. Auto-heal will attempt recovery and fail — the test stays failed. This is the correct outcome: auto-heal absorbs UI drift; it does not mask genuine regressions. Pull the failure bundle (`testsprite test failure get `) to triage what actually broke. ## Credits | Action | Cost | | :------------------------------- | :------------------------------------------------------------------------------- | | Fresh `test run` (frontend) | 0.5 credits per test executed | | Fresh `test run` (backend) | 0.2 credits per test executed, including any expanded dependency-closure members | | `test rerun` (frontend) | 0.5 credits — billed the same as a fresh run | | `test rerun` (backend) | 0.2 credits per test in the closure — billed the same as a fresh run | | `test rerun` — auto-heal engages | An additional small amount of credit, on top of the rerun charge | Reruns are billed identically to a fresh run — there is no discounted "replay" tier. (Legacy V2 accounts: a clean verbatim frontend rerun remains free.) When your credit balance is insufficient, the command exits with code 12. Top up credits in the Portal under Settings → Billing. See the full list of exit codes and what each one means. ## Where to Go Next Trigger fresh runs, override the target URL, and understand the full exit-code table Pull failure bundles, step logs, and run history after a rerun Exit 12 and everything else the CLI can emit Wire the rerun loop into your coding agent's verify-fix cycle # Running Tests Source: https://docs.testsprite.com/cli/core/running-tests Trigger a test run from the CLI, wait for a verdict, and understand every exit code the runner emits. ## Trigger a run Pass a test ID to kick off a run immediately: ```bash theme={null} testsprite test run test_3a9f21c7 ``` Without `--wait`, the command returns as soon as the run is accepted by the backend — exit 0. You get a `runId` you can use later: ```text theme={null} Run queued. runId: run_5c1d9a2b status: queued ``` The test starts executing in the cloud. Come back to it with `testsprite test wait ` whenever you're ready. ## Wait for the verdict Add `--wait` to block until the run reaches a terminal status: ```bash theme={null} testsprite test run test_3a9f21c7 --wait ``` Control how long the CLI polls before giving up with `--timeout ` (range 1–3600, default 600): ```bash theme={null} testsprite test run test_3a9f21c7 --wait --timeout 300 ``` Exit codes when `--wait` is used: | Exit | Meaning | | :--- | :------------------------------------------------------------------------------------- | | 0 | Run reached passed | | 1 | Run reached failed, blocked, or cancelled | | 7 | Timeout elapsed before a terminal status — resume with `testsprite test wait ` | A timeout doesn't cost you the `runId`: even on exit 7, the CLI prints a partial object (`{ "runId": "...", "status": "running" }`) to stdout before exiting, so a script can grab the ID and resume without having captured it earlier. ## Resuming a run If a run times out or you triggered it without `--wait`, resume polling with the run ID: ```bash theme={null} testsprite test wait run_5c1d9a2b ``` `test wait` accepts the same `--timeout` flag and emits the same exit codes as `test run --wait`. When a terminal status arrives, you see the run card: ```text theme={null} runId: run_5c1d9a2b status: passed targetUrl: https://app.example.com codeVersion: v4 startedAt: 2026-06-15T10:02:11Z finishedAt: 2026-06-15T10:04:38Z steps: 12 passed, 0 failed dashboard: https://www.testsprite.com/dashboard/tests/proj_8f0f6/test/test_3a9f21c7 ``` ## Interrupting a wait vs cancelling the run Ctrl-C during any `--wait` only detaches the CLI — **the run keeps executing (and billing) on the server**. The CLI exits `130` after printing the partial `{ "runId": "...", "status": "running" }` plus a re-attach hint and a cancel hint, so nothing is lost. To actually stop the run server-side: ```bash theme={null} testsprite test cancel run_5c1d9a2b ``` The run flips to cancelled, the test is immediately re-runnable, and the test's last verdict is untouched. Already-charged credits are not refunded. Detach vs cancel, idempotent re-cancel, multi-id summaries, and the CI cleanup pattern ## Overriding the target URL By default the run hits the URL stored on the project. Override it for a single run with `--target-url`: ```bash theme={null} testsprite test run test_3a9f21c7 --wait --target-url https://staging.example.com ``` The URL must be a public `http(s)` address; `localhost` and private IPs are rejected before any network call (exit 5). What to check if a URL is rejected Testing against a `localhost` target requires the MCP Server and its built-in tunnel, not the CLI. The CLI is designed for cloud-accessible environments and CI. ## Running a whole suite Run every test in a project in a single wave-ordered batch: ```bash theme={null} testsprite test run --all --project proj_8f0f6 --wait ``` `--all` requires `--project`. It triggers every test in the project (each billed like any run) and respects dependency waves derived from `--produces` / `--needs` annotations set at backend test-creation time — producer tests run before the consumers that depend on their output variables, with teardown tests last. How to declare `--produces` / `--needs` annotations Narrow the batch with a name substring filter: ```bash theme={null} testsprite test run --all --project proj_8f0f6 --filter "checkout" --wait ``` Control how many runs execute simultaneously: ```bash theme={null} testsprite test run --all --project proj_8f0f6 --wait --max-concurrency 10 ``` | Flag | Type | Default | Description | | :---------------------- | :----- | :------ | :------------------------------------------ | | `--all` | bool | — | Run all tests in the project (wave-ordered) | | `--project ` | string | — | Required with `--all` | | `--filter ` | string | — | Case-insensitive name filter | | `--max-concurrency ` | number | 50 | Max simultaneous triggers (1–100) | `--all` runs every test in the project in wave order. A project holds a single type of test — frontend or backend, set by `--type` at `project create` — so don't mix both in one project. For backend projects, producers run first and teardown tests last, so dependencies are always satisfied. If a batch is served by the legacy backend-only engine, frontend tests come back in `skippedFrontend` with an advisory — trigger those individually with `testsprite test run `. ## Run status and exit codes Every `test run` and `test wait` exits with a code your script or agent can branch on: | Exit code | Meaning | | :-------- | :------------------------------------------------------------- | | 0 | passed (or run queued without `--wait`) | | 1 | failed, blocked, or cancelled | | 3 | Auth error — check your API key or scopes | | 4 | Test or run not found | | 5 | Validation error — bad flag value or URL rejected | | 6 | Conflict — this test already has a run in flight | | 7 | Timeout — resume with `testsprite test wait ` | | 10 | Transport failure — retriable | | 11 | Rate limited — honor Retry-After | The full exit-code table, including codes for other commands Exit 6 means a run is already in flight for this test. Wait for it to finish (or use `testsprite test wait `) before triggering another. ## The dashboard link When a run completes, the CLI prints a `dashboard:` line in text mode and includes `dashboardUrl` in JSON output (when available). That link deep-dives directly into the Portal run view — steps, video recording, analysis, and the full failure bundle — without any manual navigation. ```bash theme={null} testsprite test run test_3a9f21c7 --wait --output json | jq '.run.dashboardUrl' ``` The `dashboardUrl` is absent under `--dry-run`. ## Where to Go Next Inspect steps, pull failure bundles, and navigate run history Replay a saved script, or let AI repair UI drift automatically Complete exit-code reference for scripts and CI pipelines Wire the CLI into GitHub Actions or any pipeline # Installation Source: https://docs.testsprite.com/cli/getting-started/installation Install the TestSprite CLI and sign in in under 2 minutes. ## Prerequisites Before installing, make sure you have: * **Node.js 20.19+, 22.13+, or 24+** (the odd-numbered releases 21.x and 23.x are not supported) * A **TestSprite account** — [Sign up for free ](https://www.testsprite.com/auth/cognito/sign-up) * A **TestSprite API key** (you'll create one in the next section) Run the following in your terminal: ```bash theme={null} node --version ``` The CLI requires **20.19+**, **22.13+**, or **24+** — the odd-numbered releases (21.x, 23.x) aren't supported. If your version doesn't fall in one of those ranges, download a newer release from [nodejs.org ](https://nodejs.org/). The CLI also checks the Node version at startup and exits with a clear message if it's too old. Sign in to your TestSprite dashboard, navigate to Settings → API Keys, and click Create new key. The key is shown exactly once — copy it before closing the dialog. If you lose it, just create a new one. See [API Keys](/web-portal/admin/api-keys) for the full walkthrough. ## Install Install the CLI globally with npm: ```bash theme={null} npm install -g @testsprite/testsprite-cli ``` Alternatively, run it without installing using npx: ```bash theme={null} npx @testsprite/testsprite-cli --version ``` Verify the installation: ```bash theme={null} testsprite --version ``` ```text theme={null} 0.3.x ``` The CLI's source code, releases, and issue tracker live on [GitHub](https://github.com/TestSprite/testsprite-cli). ## Get Your API Key 1. Sign in to your [TestSprite dashboard ](https://www.testsprite.com/dashboard). 2. Navigate to Settings → API Keys and click Create new key. API Keys page with the Create new key button 3. Copy the key — it is shown **once only**. If you lose it, delete the key and create a new one. API Key Created dialog with the copy button New and grandfathered keys both default to the scopes the CLI needs: `read:me`, `read:projects`, `read:tests`, `write:tests`, and `run:tests`. ## Sign In Run `testsprite setup`. It prompts for your API key, verifies it against the platform, and installs the verification skill into your project's agent configuration — all in one step: ```bash theme={null} testsprite setup ``` ```text theme={null} TestSprite API key: ******** TestSprite initialized. profile: default env: production email: alice@example.com scopes: read:me, read:projects, read:tests, write:tests, run:tests agent: claude (installed) Next steps: # 1. Create your first project (frontend example) — prints a projectId testsprite project create --type frontend --name "My App" --url https://your-app.com # 2. Generate tests: ask your coding agent (the testsprite-onboard skill is # installed), or create one yourself, then run them: testsprite test run --all --project # Manage installed agent skills testsprite agent list testsprite agent install --target= # re-install or install additional targets ``` `setup` chains credential configuration → identity verification → agent skill install. You only need to run it once per project. Pass `--agent ` to pick a different coding agent (default `claude`). Add `--no-agent` to configure credentials without installing the agent skill: ```bash theme={null} testsprite setup --no-agent ``` Then confirm the identity bound to the key: ```bash theme={null} testsprite auth status ``` `testsprite setup` is the single onboarding command — it configures credentials, verifies them, and installs the agent skill in one pass. Add `--no-agent` when you only want credentials. ## Verify After signing in, confirm the active profile: ```bash theme={null} testsprite auth status ``` ```text theme={null} userId u_01abc... name Alice Example email alice@example.com keyId key_01xyz... env production scopes read:me, read:projects, read:tests, write:tests, run:tests ``` If any scopes are missing, the CLI prints a `note:` line telling you which commands will be blocked. ## Using It in CI In a CI environment, set `TESTSPRITE_API_KEY` as a secret and run setup non-interactively: ```bash theme={null} TESTSPRITE_API_KEY=$TS_KEY testsprite setup --from-env --yes ``` `--from-env` reads the key from the environment instead of prompting. `--yes` accepts all defaults without interactive prompts. The CLI never accepts the API key as a positional argument or writes it to logs. The full CI setup — pipeline examples and exit code handling ## Where to Go Next Create a test, run it, and read the failure bundle end to end Profiles, env vars, scope errors, and rotating keys Projects, tests, runs, and the object model Pipeline examples and non-interactive setup # Overview Source: https://docs.testsprite.com/cli/getting-started/overview The verification layer for the agentic coding era — TestSprite CLI puts the full testing platform in your coding agent's hands. ## What is the TestSprite CLI? **The verification layer for the agentic coding era.** TestSprite is the AI testing platform 100,000+ teams use to test their software — frontend and backend — in the cloud, against the live product, not mocks. The CLI puts that platform in your coding agent's hands: structured output it can parse, exit codes it can branch on, and one self-consistent failure bundle it can act on — no dashboard scraping.