A Better Vocabulary for Testing
Hacker News

A Better Vocabulary for Testing

Comments

Contemporary vocabulary for talking about software testing has bothered me for a long time.

The concept of a unit in unit test is not really defined, integration tests are integrating arbitrarily scoped modules, the concept of an E2E test makes sense, but it very much underspecifies the actually tested behavior as well as methods for testing it, are you using automated agents, human supervision, predefined set of actions, random exploration... it's all sweeped under the rug.

On the other side you have your stress tests, load tests, peneration tests etc. but there is no proper classification in the wild, it all seems to be based on "vibes".

A short web search reveals a "level" based classification as seen below, as well as an "ad" from Atlassian that gives you 7 "types" of software tests.

Can we go beyond this, definitely. Should we, I think yes. We should have a vocabulary that is actionable, you should be able to upgrade or downgrade your tests based on the availability of resources, there should be a recipe for the available set of options given your situation.

Of course, this article will be very much biased towards a certain category, random testing which I happen to be working on for a while now.

My view is as follows, we can roughly create 5 different categories, some of which are orthogonal to each other, some of which "branch" within.

Scopes of Testing

The first category is a the scope of the test unit, this is similar to what the classic unit/integration/e2e testing classification tries to vaguely define, but I think it fails at it.

There are two axes to the scope, one of them is based on the constructs of the programming language we use, we might test a function, a class/struct/type, a module, or a system.

The secondary axis is if the scope is a stateful, or a stateless one. You might have stateful functions, and you might have stateless systems, although we expect most functions to be stateless and most systems to be stateful.

It should be noted that these scopes also vary a lot, one could argue that the entry function of a large system is the same as the system itself, but here is why the scopes are useful; When talking about testing anything, we typically talk about inputs and outputs, but the space of inputs are actually very different.

Functions, especially stateless ones, consume arbitrary structures in your programming language. Systems receive interactions or directives, which is typically some serialized piece of data that is parsed and validated.

This actually allows system testing to be much more restrictive in the list of inputs because it includes a natural discriminator that can shrink the input space, while function signatures in programming are very often too relaxed, the input type of the function includes many potential inputs that the function will never consume.

When state comes into play, your testing can be made isomorphic to the previous one by turning a test of function f with state s consuming input x and producing output y and transforming the state to s' into another stateless function g that either produces initial the state s or the state pair (s, s') becomes a part of the input-output.

This type of state(ful/less)ness transformation depends on the complexity of your system, because testing an input requires you to produce it. Simple states can be turned into inputs that you can construct, but creating a complex state that represents a realistic workload of your system is non-trivial, and for many systems outright impossible.

In such scenarios, instead of transforming your test to be stateless, you treat your system as a simulation. You pass in messages that contain directives or interactions based on the public API of your system and get answers or confirmations in multi-message scenarios, you can check intermediate states of the system as well as the initial and final ones, ensuring your invariants, preconditions and postconditions hold.

Similarly for making the testing of your functions more tractable, you can employ Design by Contract (DbC) which is natively supported in C++26, D, Kotlin, and Racket, with library support in Python and Rust, and many others I probably haven't heard.

In DbC, we can write the assumptions we make on the inputs to the functions, which might come up as unchecked pointer dereferences or code comments promising that such inputs are not possible, as annotations to our functions, which can be checked on runtime, and sometimes even compile time through static analysis.

DbC allows us to constraint the input space to the functions, which as I said are at many times much larger than the reality. Imagine creating a dictionary of items you read in your system, along with a list of keys that were created alongside the initial hashmap.

To the view of your programming language, every lookup in your hashmap is fallible, when in reality we know all of them should succeed. This creates a tension between you and the programming language, where you feel like you are fighting with it.

Using DbC, more expressive type systems, and better domain modeling solves much of these issues.

Purposes and Hypotheses of Testing

Testing, at some level, evaluates or measures "something". It creates a hypothesis that can be falsified by running the test, and the purpose is to falsify this hypothesis.

There are different domains that we can express some hypotheses on; the simplest one is functional correctness, which evaluates if the function produces the right output given some set of inputs.

There are other domains such as time of execution, which we can test the performance of our system under test, or security, which we can check if the system is resistant to potential attack vectors.

Most large scale systems are distributed systems that are bound by the laws of physics that can cause delays, drops, faults in the network, and even single machine systems experience file system fauls, disk corruptions etc. which we can test our system's resilience against.

The concept of chaos testing mostly refers to testing the robustness of a system in the case of unexpected faults.

Based on the domain of interest, such purposes or hypotheses varies a lot. In a security scenario that tests a potential timing side channel, the hypotheses is that the test will constuct an environment in which the attacker will be able to extract information out of the time to execute two different inputs.

When testing for accessibility of a system or a webpage, the hypothesis is that a user with some disability that prevents them from leveraging the full set of system interactions will not be able to perform an action they would need or want to.

Each test, of course, requires a domain specific oracle based on the scenario.

Oracles of Testing

Once we have an hypothesis at hand, we need an oracle for it. The natural oracle is a human.

Although it's an understatement to say that this is the job of QAs in software companies (they also program oracles by creating automated scenarios), it's also very much they excel at. They hand-craft scenarios to test against the system in the hopes of discovering an "unusual" or "unwanted" scenario.

Some of these scenarios may be more formal, for example it is commonplace that your software should not be stuck or unresponsive at any moment, but some others will be based on intuitions like "this page transformation is weird compared to the others".

I like to call this type of testing where the oracle is the judgement of a human user "tinker testing". Although it's not typically included in the discussions of testing, tinker testing is a major part of programming, because a traditional software test is ultimately a codification of tinker tests.

If you are not doing test-driven development (I mostly don't), it is very probable that you start by implementing a piece of functionality, create a small test where you can change the parameters very easily, maybe by changing them by hand if the compilation is very fast, or by using some functionality for reading inputs from the command line or web user interface.

We play around with the artifacts the software produces for understanding it better, because not every piece of functionality in software is intentional, or explicit. Many code has unintended consequences, which is partly why we test it.

The tinkering allows for:

  • (1) checking that the artifact conforms to the intended consequences, and
  • (2) finding out what kind of unintended consequences we ended up sneaking into the artifact.

The more "dense" the code is, the higher the possibility of introducing unintended consequences, which increases the requirement for tinker testing as well as testing in general, including applying TDD, but that's a discussion for another day.

The second type of oracle is examples. When you write your unit tests with a concrete set of values to say add(3, 5) == 8, you are doing example-based testing, as you were doing tinker testing when you were typing 10 + 32 to your calculator for checking that it indeed tells you the result is 42.

You can write example-based oracles for all kinds of hypotheses, even though we mostly think of them useful for functional correctness. You can say measure the time of the execution of a function for specifying upper bounds such as time(f(1337) < 0.5), or time(sort([])) < time(sort[10, 9, 8, 7, 6, 5, 4, 3, 2 1]).

You can check if your SQL sanitizer is indeed catching specific SQL injections you can conjure. However complex the scenario is, there is typically a way to create an example for it.

Some approaches has taken this idea very far and that is why we have expect testing, golden testing, or snapshot testing in the testing literature today, where the testing workflow evolves by verifying that the output conforms to the expectation of the tester for the given user, and the testing framework saves the snapshot to a file by codifying the current state of the program as the correct behavior.

Although the primary use case for expect testing seems to be catching regressions, it also gives us a way to create more examples very easily, which actually possesses many hidden or unexpressed semantics of our program within, so they make for interesting oracles.

For more on expect testing, this blog post from Jane Street is a nice read.

A third type of oracle is inline assertions, which actually ties back to DbC I've talked at the end of the scopes. A precondition is an assertion at the beginning of a block, a postcondition is an assertion at the end of a block, an invariant is an assertion that should hold both at the beginning and end of the block, although it may be broken in the middle.

An inline assertion is an assertion you can put anywhere in your program, there are companies in high-assurance part of the industry that advocate for aggressive use of inline assertions even in production, I advise everyone to read the Tiger Style documentation, they have very good principles for writing robust and safe software.

Inline assertions are not really coupled with a testing method by themselves, as opposed to the other oracles I mentioned so far, both of which are oracles and testing methods at the same time.

Tinker Testing couples human intuition and tinkering as the testing method with human judgement as the oracle, and Example-Based Testing couples input-output examples as both the oracle and the testing method, because the user is producing the input (the method of testing) alongside the expected output (the oracle).

Expect testing somewhat couples these approaches, the user still provides the input, but the oracle is now their judgement at first, and the state of the program at the time of snapshotting later.

Inline assertions can be used alongside other methods of testing, because they are a method of specification but not a method of producing inputs for the system under test.

The fourth type of oracle is properties, which brings us to Property-Based Testing. Properties are universal statements that talk about the code under test in abstract terms rather than concrete inputs in the case of example-based testing.

Some properties may be global invariants, which means they are like inline assertions you check at the end of each unit of execution. Examples of these are "program never crashes" in many languages (maybe not Erlang I guess?), or "never dereference a null pointer", or if you are running a physics simulation it may be that you want to conserve momentum at each time step, or you may have any other type of global consistence metric based on your domain.

Other properties are more local, they are metamorphic properties that relate the outputs of two programs/functions to each other. For instance, the associativity of addition tells us that a + (b + c) = (a + b) + c (which doesn't hold for IEEE-754 floats, so be careful), or the UNION of two SQL queries should be a subset of their UNION ALL.

We have other canonical properties like roundtrip properties which tells us that two functions that are the inverse of each other result in the original input, for instance a lossless compression algorithm should always have the property that decompress(compress(bytes)) = bytes, or adding and subtracting the same quantitity should result in the original quantity a + x - x = a - x + x = a, which also amounts to some sort of cancellation effect.

Compatibility is another sort of canonical property, different procedures that are functionally equivalent should always return the same results given the same inputs, so all sorting algorithms should always return the same result on the same list quicksort(l) = bubblesort(l), or parallelizing some functionality should not change its observable behavior par(f, x) = f(x), or your very complex distributed system should provide the same observable behavior as its simpler, single-node version.

This type of compatibility, sometimes called differential testing, or other times model-based testing, is immensely

Read on Hacker News ↗ ← Back to News

Comments

No comments yet. Start the discussion.