More Testing Terminology

Posted on by Chris Warburton

This is mostly copypasta from an answer I gave on StackOverflow about how to avoid mocks when unit testing. It’s a continuation of the argument I gave in an earlier blog post.

I think the subject of automated testing suffers from conflated and co-opted terminology, which causes people to talk past each other.

For example, the questioner asks the following:

Should I be writing only integration tests when there is dependency, and unit tests for pieces of code without any dependency?

I think most people would answer this question by saying something like (ideally, modulo common sense, etc.):

When there is no dependency, unit tests are sufficient and mocks aren’t needed; when there is dependency, unit tests may need mocks and there should also be integration tests.

Let’s call this Answer A, and I’m going to assume that it’s a relatively uncontroversial thing to say.

However, two people might both give Answer A, but mean very different things when they say it!

When a person who the questioner terms a “classicist” says Answer A, they might mean the following (Answer B):

Functionality that is internal to the application (e.g. a calculation which performs no I/O) doesn’t need integration tests, and its unit tests don’t need mocks. Functionality with some external dependency (e.g. a separate application like an RDBMS, or a third-party Web service) should have integration tests, and if it has unit tests they may need the external interactions to be mocked.

When others (let’s call them “mockists”) say Answer A, the might mean the following (Answer C):

A class which doesn’t call methods of another class doesn’t need integration tests, and its unit tests don’t need mocks. Classes which call methods of other classes should mock those out during their unit tests, and they should probably have integration tests too.

These testing strategies are objectively very different, but they both correspond to Answer A. This is due to the different meanings they are using for words. We can caricature someone who says Answer A, but means Answer B, as saying the following:

We can caricature someone who says Answer A, but means Answer C, as saying the following:

These are very different meanings, but the relationships between B’s meanings and between C’s meanings are similar, which is why both groups of people seem to agree with each other about Answer A (e.g. their definitions of “dependency” and “integration test” differ, but both have the relationship “dependencies should have integration tests”).

For the record, I would personally count myself as what that questioner calls a “classicist” (although I’d not come across that term before); hence why the above caricatures are clearly biased!

In any case, I think this problem of conflated meanings needs to be addressed before we can have constructive debates about the merits of one approach versus another. Unfortunately every time someone tries to introduce some new, more specialised vocabulary to avoid the existing conflations, those terms start getting mis-used until they’re just as conflated as before.

For example, “Thought Leader X” might want to talk about physical humans clicking on a UI or typing in a CLI, so they say “it’s important to describe how users can interact with the system; we’ll call these ‘behaviours’”. Their terminology spreads around, and soon enough “Though Leader Y” (either through misunderstanding, or thinking they’re improving the situation), will say something like “I agree with X, that when we design a system like the WidgetFactory class, we should use behaviours to describe how it interacts with its users, like the ValidationFactory class”. This co-opted usage spreads around, obscuring the original meaning. Those reading old books and blog posts from X may get confused about the original message, and start applying their advice to the newer meanings (after all, this is a highly regarded book by that influential luminary X!).

We’ve now reached the situation where “module” means class, “entity” means class, “unit” means class, “collaborator” means class, “dependency” means class, “user” means class, “consumer” means class, “client” means class, “system under test” means class, “service” means class. Where “boundary” means “class boundary”, “external” means “class boundary”, “interface” means “class boundary”, “protocol” means “class boundary”. Where “behaviour” means “method call”, where “functionality” means “method call”, where “message send” means “method call”…