The Test Pyramid

The test pyramid is a visualization which kinds of tests need to be used to test a tool successfully. It also shows clearly how much effort should be invested in which kind of test. All kinds of tests, except manual tests are fully automated. This means, you push one button or execute one shell command to run all tests. The tests are also verified automatically.

Tests

Unit Tests

Unit tests are the basis of the test pyramid. A unit is a small part of code, focussing on a specific task, for example, a single class or function. Unit tests are written by developers and are targeting developers. Their primary purpose is to check if the application logic is working correctly, and everything is executed as expected. They should cover the whole code and check if every line is working correctly. Ideally, Unit tests are written even before the normal code is written. This technique is called “Test Driven Development (TDD)”. Here you can find a more detailed explanation of TDD. Developers should always execute unit tests while they write code. They should be able to do a small change and run the unit tests to see if something broke. For this reason, unit tests need to be very fast. Otherwise, developers need to wait too long for the tests to complete. Since unit tests focus on single units, other units are mocked.

Integration Tests

Integration tests test the interaction of multiple units (classes and functions). They don’t test the logic, since the logic was already tested in the unit tests. To check the correct interaction, they verify that data is passed correctly between the individual units. Integration tests should not cover every single line of code. Since they test multiple components, they run slower than unit tests and can not be executed whenever a developer changes a line of code.

UI Tests

UI tests are tests done on the UI level. A UI can either be a GUI or a command-line interface. UI tests check if the UI works correctly, meaning inputs are successfully processed and data is displayed correctly. They run against a complete application with no units mocked. For this reason, UI tests are slow to execute. Since UIs can change quickly, UI tests can break easily, so they should cover less code than integration tests and not test any logic. UI tests are fully automated. GUIs for visual effects pipelines are mainly written using Python bindings for the Qt framework. For Qt, there is a pytest plugin pytest-qt to provide functionality specifically for UI tests.

Manual Tests

Manual tests verify that the application is working correctly when used by humans. Humans and especially artists tend to produce unexpected behavior like weird and wild clicking and random key pressing. Since humans will use our software, we need to make sure that these unexpected inputs don’t crash our application. Manual tests should be only done from time to time and very limited. Otherwise, all the benefits of automated tests are lost.

Anti-pattern: The Inversed Test Pyramid

The Inverted test pyramid is precisely the opposite of how the test pyramid should be applied.

In the inverted test pyramid, the tests are mainly focusing on the UI. Because they focus on the UI, they run very slow and can not be executed every time a line of code changes. They tend to break often because of UI changes. Don’t use the inverted test pyramid.

Example

To see how the test pyramid is applied to a visual effects tool, see the section How is the test pyramid applied? in the example Arnold Subdiv Manager.