The Testing Pyramid

Agile has totally changed the focus of testing and introduced automation to support its development practices. I have been involved with software development teams for nearly thirty years. Over that time I have seen many different methods and practices come and go but testing has remained focused around manual testing. That is until Agile software development arrived.

An Agile iterative approach to software development means you have to test all your software all the time. Ensuring what you have just built has not broken a previous written piece of functionality. Agile automates testing at all layers of the application. This approach to testing is fast overtaking the traditional manual approach. Automated tests that previously existed were focused on testing the front end, the most volatile part of the application. They were also written after the application was built therefore not giving the short feedback loop we use in Agile software development. The Testing Pyramid puts a different testing emphasis on each of the application layers and focuses efforts at the beginning of the development cycle rather than at the end.

The Testing Pyramid

Looking around the web you will see various implementations of the Testing Pyramid with different names for the levels and implementing different types of tests. The one I use is the basic three level pyramid – Unit Tests, Service Tests and UI Tests. As with any pyramid, each level builds on the solidity of the level below it.

From technical view point one can look at these three levels as small, medium and large. Small tests are discreet unit tests that use mocks and stubs to collaborate with other objects and are typically written in one of the xUnit frameworks. Service level tests are the medium sized tests and interface with one other system, typically a database or a data bus. Large tests, UI tests collaborate with many sub-system parts they support the end-to-end scenarios.

Personally I look at the levels in a functional way:

  • UI Tests: Does the software work as expected by the end user?
  • Service Tests: Does the software meet the acceptance criteria on the user story?
  • Unit Tests: As a developer does the software work as I expect?

Unit Tests

Unit tests are the foundation of the Testing Pyramid. This is where the bulk of tests are written using one of the xUnit frameworks – for example, JUnit when developing in Java. They ask the question “Are we building the product right?”. When writing software I like to take a Test Driven Development (TDD) approach. TDD is a design approach to development where tests are written first and code written to support the test. There are a number of benefits to taking this approach:

  • High test coverage of the written code
  • Encourages the use of Object Orientated Analysis and Design (OOAD) techniques
  • Allows you to move forward with confidence knowing the functionality works
  • Debugging is minimised because a test takes you straight to the problem
  • Developers take more responsibility for the quality of their code
  • Because it is written to support specific tests, the code works by design not by coincidence or accident

Service Tests

I see Service Tests as supporting the acceptance criteria on the user story. They ask the question “Are we building the right product?”. When writing these tests, I like to take advantage of the Given/When/Then BDD format of the acceptance criteria and use one of the BDD test frameworks, typically Cucumber. I only adopt specification by example that is use real world examples in the acceptance criteria. This approach gives a number of benefits;

  • Assurance that all stakeholders and delivery team members understand what needs to be delivered through greater collaboration
  • The avoidance of re-work through precise specifications and therefore higher productivity

UI Tests

The User Interface is the most volatile part of the application and should not contain any business logic. For these reasons the least emphasis on automated testing should be here. That does not mean there is no testing, I like to automate key user journeys through the system using one of the UI testing frameworks e.g. WebDriver. UI testing demonstrates that all subsystems are talking to each other.

I use manual testing for the look and feel, checking the UI acts as expected and exploratory testing to find those hidden nuances.

Test Delivery

Unit tests and Service tests should be delivered in the iteration by the delivery team. As part of their development practices developers write Unit tests when building the functional code. Ideally a test first, TDD approach should be used.

My conclusion?

The Testing Pyramid inverts the traditional approach to testing. The focus is at the beginning of the development process with the developer taking responsibility for quality of their code. This is a very different way at looking at the problem from the traditional approach where code is handed over to the tester and they are assumed responsible for the code.

The early identification of defects gives two major business benefits;

  • Issues are discovered early in the development process reducing the cost of defect fixing associated with late discovery
  • Issues are identified and resolved early therefore negating the need to postpone the production release through late identification of issues

References

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

http://martinfowler.com/bliki/TestPyramid.html

http://www.agilecoachjournal.com/index.php/2014-01-28/testing-2/the-agile-testing-pyramid/