How to Write Postman Performance Test Scripts That Scale
DEV Community

How to Write Postman Performance Test Scripts That Scale

Originally published on the Postman blog.

Your load test passed. One of its requests returned a 404.

If you already have Postman collections and want to turn them into load tests, this is the trap to know about. Postman doesn't count an HTTP status as a failure on its own. A green run can hide broken requests, and a script that works for one user can quietly misbehave at 50.

This guide shows how to get the script right at one virtual user (VU) first: real assertions, a single sign-in, steady pacing and a p95 gate in CI. Only then do you scale.

A Postman performance test is only as good as the script behind it. A collection that works for one request can quietly misbehave when 50 virtual users run it at once: a variable that was never set, an assertion that only holds for one record, a sign-in that every user fires in the same second.

This post walks through building that script. You get a collection, chain requests, write assertions that mean something under load, sign in and clean up with setup and teardown, and feed the test data. Just as important, you debug it before you add load. Once it runs cleanly, you can try it in the app and then take it to CI with the Postman CLI or Postman Cloud.

The workflow at a glance

  • Get a collection: import your OpenAPI spec, or record it with the Postman proxy, fork one, or write it.
  • Script it: chain values between requests and control the pace.
  • Assert on correctness and on a response-time budget.
  • Sign in and clean up with setup and teardown collections.
  • Feed it data with a data file or a dataset.
  • Debug it at one user, before you add load.

You will loop between Debug and Script until a one-VU run is clean, and only then run it at scale.

What a performance test script is

There is no separate scripting language for performance tests. A performance test runs the requests in your collection (or one folder) in order, in a repeating loop, for every VU at once, for a set duration. Your pre-request scripts, post-response scripts and pm.test() assertions run on every pass of that loop, the same way they run in a functional run.

That has two consequences:

Your script runs thousands of times. A slow or noisy script is multiplied by every request, every VU and every iteration. On a local run, heavier scripts reduce the number of VUs your machine can simulate.

Many copies run at once. Anything that assumes it is the only user (a shared record, a one-time sign-in, a counter) behaves differently when it is not.

Prerequisites

The Postman desktop app, where you also run a local performance test, and a workspace.

Docker, to run the demo app.

For Step 5: a paid plan if you want datasets or data files in a performance test.

Set up the demo app

The examples use Swagger Petstore, an open source (Apache-2.0) sample REST API with an OpenAPI spec. It runs on your own machine, which matters: never point a load test at a shared public service, and never at production.

The app listens on 8080 inside the container, and the setup maps it to 18080 on your machine. Port 8080 is one of the most commonly used ports, and a clash fails quietly (see Step 6, pointer 4).

To check that 18080 is free first (macOS and Linux), run the command below. No output means the port is free.

lsof -nP -iTCP:18080 -sTCP:LISTEN

Give the app a few seconds to start. The OpenAPI spec is at http://localhost:18080/api/v3/openapi.json.

Four things are worth knowing before you start:

  • Pin the tag. The README suggests :unstable, which is a March 2023 build. 1.0.28 is the current release.
  • Timings are not benchmarks. The image is built for amd64 only, so on Apple Silicon it runs under emulation.
  • State lives in memory. Restart the container for a clean slate.
  • Auth is declared but not enforced. The spec lists OAuth2 and an api_key header, but the server accepts requests without either.

Step 1: Get a collection

A performance test needs one thing: a collection (or one folder of one). The rest of this post uses a single collection, Petstore Load, built from the app's OpenAPI spec.

Import the OpenAPI spec

If your API has an OpenAPI spec, importing it is usually the fastest route, and it doubles as a check that the spec matches reality. In the app:

Click Import. Add Petstore's spec: paste http://localhost:18080/api/v3/openapi.json, or save it with curl -o openapi.json http://localhost:18080/api/v3/openapi.json and drag the file in. Leave the import option as it is (OpenAPI 3.0). Press Enter to import, then rename the new collection to Petstore Load.

Petstore's spec has 19 operations, so you get 19 requests grouped into folders by path, each with saved examples. A collection generated from an OpenAPI 2.0, 3.0 or 3.1 spec can be kept in sync with the spec afterward, so when the spec changes you update the collection instead of rebuilding it.

Prefer the command line? This builds the same kind of collection as local files (add --folder-strategy Tags to group by tag):

postman spec generate collection ./openapi.json -n "Petstore Load"

Fix what the import gets wrong

Set the base URL. Petstore's spec declares its server as the relative path /api/v3, so the collection's baseUrl has no host. Change it to http://localhost:18080/api/v3.

Check the request bodies. The generated "Add a new pet" request uses the form-encoded body variant. Step 2 replaces it with a clean JSON body.

Watch the order. Requests run in folder order, which is alphabetical here, so the sign-in request runs last. Step 4 moves sign-in into a setup collection, which runs before everything else.

Add assertions. A generated collection has no tests. Step 3 adds them.

Trim it to the requests you will test

Load-testing all 19 requests is not the goal. Keep the four that make up one realistic flow, put them in a folder called Pet Lifecycle, and delete the rest. Sign-in is not among them: it moves to its own setup collection in Step 4.

Run it once before you load it

Send the requests once before you add load. A performance test repeats the collection in a loop for every VU, so anything that should happen only once will happen thousands of times. Move one-time work into a setup collection (Step 4).

Read the result with suspicion. With baseUrl fixed and nothing else changed, the four-request collection reports 4 requests, 0 failed. Yet one of those four, GET /pet/4808, came back 404 Not Found. Nothing failed because nothing was asserting anything. Postman does not treat an HTTP status as a failure on its own, so a collection without assertions can only fail on a transport problem. Add assertions (Step 3) before you trust a green run.

Other ways to get a collection

  • A cURL command or HAR file: paste it into Import, or run postman collection import <curl or file.har> --name "...".
  • Real traffic from a client: the Postman proxy (Tools โ†’ Proxy) records what a client sends and saves it as a collection.
  • A collection someone else built: fork it or import the file.
  • From scratch: + โ†’ Collection, or postman collection new "Orders API".

Step 2: Script it

Pre-request and post-response scripts run under load exactly as they do locally, so your functional scripts carry over. Every script runs once per request per VU, so keep them lean.

Know your variables first

Postman has five variable scopes. From broadest to narrowest, they are global, collection, environment, data and local. When the same name exists in two scopes, the narrowest one wins.

Who else sees a variable's value? Collection, environment and global variables are local to your copy of Postman by default. The value you type is used when you send requests, and it is not synced to the Postman Cloud. A variable stays yours until someone with Editor access chooses to share its value. Changing a variable from a script changes your local value only.

Don't confuse teammates with virtual users. During a performance test, each VU runs with its own copy of the collection variables. A value one VU sets (a token, a counter, an iteration start time) is never visible to another VU. That is why per-VU sign-in and the pacing timestamp below are safe. It is also why collection variables cannot coordinate VUs.

The shared value matters for load tests in two ways:

Cloud features use the shared value, not your local one. That includes cloud performance runs, scheduled runs, monitors and the Postman CLI. A variable that is only local works on your laptop and comes up empty when the same collection runs in the cloud or in CI. Share the values your test reads before you run it anywhere but your own machine.

Shared values are not private. Never share a password or token as a plain value. Mark it as a secure variable, which stores the value in Postman Vault.

Rule of thumb: Put test settings (think time, pacing, flags) in shared collection variables. Put the base URL in an environment. Put per-iteration input in a data file. Put credentials in secure variables.

Make the settings variables, not code

Built-in think time and pacing settings are not part of the performance test configuration yet, and the Postman team is working on them. Until then, scripts can do the job. Put every tunable in a collection variable with a sensible default, and have the scripts read it. Document the knobs in the collection description.

Chain values between requests

The Petstore flow is add a pet, get it, find pets by status, and delete it. Each request needs the ID the first one chose, so give every iteration its own pet ID. That way concurrent users don't step on one another's data.

In a pre-request guard, use pm.execution.skipRequest(). In a pre-request script, pm.execution.setNextRequest(null) does not stop the request it runs in. In a post-response script, setNextRequest(null) is the right call, because the request has already been sent.

Later requests read the same variables, such as GET {{baseUrl}}/pet/{{petId}} and DELETE {{baseUrl}}/pet/{{petId}}.

pm.test("add pet returns 200", () => { pm.response.to.have.status(200); });
pm.test("add pet echoes the id we sent", () => { pm.expect(String(pm.response.json().id)).to.eql(pm.collectionVariables.get('petId')); });

Sign in once, and ramp the users up

If every VU signs in at the same instant, you are testing your login endpoint's tolerance for a stampede, not the flow you care about. The Postman CLI has two built-in answers. Both are in Postman CLI 1.75.0: check postman performance run --help for --setup-collection and --load-profile.

Sign in once, for the whole test. Put the sign-in request in a setup collection. It runs once before any VU starts, and Postman passes the environment it updated on to every VU. No sign-in request is left in the loop, so there is no stampede. Step 4 shows the details.

Ramp the VUs up. With the CLI, the default load profile is ramp-up: --load-profile ramp-up --base-vu 2 --peak-vu 50 --duration 10 --load-segments 1m,3m. That is 2 VUs for the first minute, a climb to 50 over the next three, and 50 until minute 10. The base VUs still start together, so keep the base small.

When one token is not enough. If each VU needs its own session, or the token can expire mid-run, sign in once per VU inside the loop:

pm.test("sign in returns 200", () => { pm.response.to.have.status(200); });
// Petstore answers with plain text: "Logged in user session: "
const match = pm.response.text().match(/session:\s*(\S+)/);
if (match) {
  pm.collectionVariables.set('accessToken', match[1]);
} else {
  console.log('Sign in did not return a session, ending this iteration.');
  pm.execution.setNextRequest(null);
}

Add think time

Without think time, every VU fires its next request the instant the last one returns. Add a random pause in each request's post-response script:

const min = Number(pm.collectionVariables.get('THINK_TIME_MIN_MILLIS')) || 0;
const max = Number(pm.collectionVariables.get('THINK_TIME_MAX_MILLIS')) || 0;
if (max > 0) {
  const delay = Math.floor(Math.random() * (max - min + 1)) + min;
  await new Promise(resolve => setTimeout(resolve, delay));
}

Because the pause is gated on max > 0, one variable turns it off for a quick smoke test.

Hold a steady pace

Think time spaces out requests within an iteration. Pacing fixes how long the whole iteration takes. Without pacing, a fast API makes each VU loop faster, so the load you send quietly grows with how well your API performs. A pacing floor makes every iteration take at least a fixed time, so the request rate stays predictable.

// First request of the loop
const pacing = Number(pm.collectionVariables.get('PACING_DURATION_MILLIS')) || 0;
if (pacing > 0) {
  pm.collectionVariables.set('iterationStartMillis', String(Date.now()));
}
// Last request of the loop
const pacingTarget = Number(pm.collectionVariables.get('PACING_DURATION_MILLIS')) || 0;
const pacingStart = Number(pm.collectionVariables.get('iterationStartMillis'));
if (pacingTarget > 0 && Number.isFinite(pacingStart)) {
  const remaining = pacingTarget - (Date.now() - pacingStart);
  if (remaining > 0) {
    await new Promise(resolve => setTimeout(resolve, remaining));
  }
}

Once pacing is on, the request rate is roughly VUs ร— requests per iteration รท pacing time. That is how you scale VUs without scaling the rate by accident.

Measure to confirm the knobs do what you think. Sleep time is not counted as response time.

Keep scripts cheap. Skip console.log on the hot path. Parse the body once and reuse it. Fail loudly on missing inputs. A thrown error counts as an error in the run.

Step 3: Add test assertions that mean something under load

Think in two layers: correctness (status code, shape) and budget (response time).

pm.test("Status is 200", () => pm.response.to.have.status(200));
pm.test("Responds within budget", () => {
  pm.expect(pm.response.responseTime).to.be.below(800
Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.