How we score the impact of an API change from 0 to 100
DEV Community

How we score the impact of an API change from 0 to 100

A risk number on a pull request is only useful if the reviewer can tell where it came from. We build Graphwake, which tells the teams that consume an API when a change in its OpenAPI spec is about to break them.

When we added an impact score to it, we wanted one you could recompute on a napkin. So every change set gets a score from 0 to 100 made of four parts, each with a cap, and the rules are fixed and shown next to every score. Here are the rules, in full.

The four parts

  • Breaking changes - Breaking changes in the change set - 10 points each - Cap 30
  • Affected teams - Consumer teams subscribed to an endpoint with a breaking change - 12 points each - Cap 36
  • People to notify - Distinct people in those teams - 1 point each - Cap 14
  • Blind spots - Removed endpoints, and breaking changes on endpoints nobody subscribes to - 10 points each - Cap 20

Each part scores the lower of two numbers: its count times its points, or its cap. The caps add up to 100.

The bands

  • 0 to 29 - Low
  • 30 to 59 - Medium
  • 60 to 100 - High

A change set with no breaking change scores 0. Nothing breaks, so there's nothing to weigh.

A worked example

Take a hypothetical change set that removes one response field and adds a required parameter on another endpoint.

  • That's 2 breaking changes: 20 points.
  • Three teams subscribe to the affected endpoints. 3 times 12 is 36, which is exactly the cap: 36 points.
  • Together those teams have 9 people: 9 points.
  • One of the two endpoints has no subscribers. That's 1 blind spot: 10 points.

20 + 36 + 9 + 10 = 75, which lands in High.

Why a blind spot counts as risk

A removed endpoint, or a breaking change on an endpoint with no subscribers, is a change whose reach is unknown. We could have scored it as zero, since nobody is subscribed and nobody gets alerted. That would reward exactly the situation you should worry about.

Counting it pushes the team to find out who is affected before merging. To help with that, Discovery suggests likely teams, with the reason for each suggestion.

What the score is for

It helps reviewers decide where to look first and who to talk to. Low means few or no teams are affected. High means several breaking changes, several teams, or blind spots, and the blind spots are where someone should go find out who calls those endpoints.

How Graphwake uses the score

The affected teams in the score are the same teams Graphwake alerts. Consumer teams subscribe to the endpoints they call, and when a change set breaks one of those endpoints, only the subscribed teams hear about it, in Slack, Microsoft Teams or email. The score tells the reviewer how wide that reach is.

With the CI step, reviewers see the score while the change is still a pull request, before the merge. On GitHub Actions and Azure DevOps, the pull request comment lists the breaking changes, the teams they affect and the score.

Example: a different pull request (#1284) as it shows in Graphwake, each part with its arithmetic.

Each score is shown with its four parts and the arithmetic, so "why did this get 75?" always has an answer on screen.

Originally published on graphwake.com.

Do you put any kind of risk number on API changes today? If you do, can your reviewers explain how it was computed?

About Graphwake

We build Graphwake, a dependency map for your APIs. It watches your OpenAPI specs, knows which teams consume each endpoint, and tells only those teams in Slack, Microsoft Teams or email when a change is about to break them. Start a 7-day free trial at graphwake.com

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.