Learning about software testing from the paper behind Karma test runnerApril 4, 2017
Karma itself originates from a university thesis, which goes into detail about the design and implementation.
1. This project does not need tests? It probably does.
The first thing you need to start testing a software project are solid reasons to do so. It sounds simple, but often automated testing seems like overkill. At first. Then the project grows and other things take priority. Finally it reaches a point where it becomes obvious that automated tests would really have helped along the way, but it is too late now. So what we need is good reasons to motivate automated testing in the first place, because they primarily pay off in the long term. Here are some of my favorites from the paper:
- Provide quick feedback enabling developers to be more productive
- Lower barrier of entry to refactoring legacy code
- Well written tests can serve as living documentation that cannot get outdated
- Enforce loose coupling and strict contracts between individual parts of the system
2. Individual test suites and cases must run in isolation
Different test suites must not share any state or data. Karma runs the actual tests inside an iframe and achieves this by refreshing the iframe completely for each test suite. A failing unit test should strongly hint to a bug within the module it tests, otherwise it is not really useful. Shared state, functions and abstractions within tests make it harder to provide this guarantee. Most testing frameworks will ship with some API to run code before and after each test. For example Mocha 2 has
afterEach methods. PHPUnit 3 calls them
tearDown. I never really thought about these before, but now it seems that they have a lot of potential for violating the isolation of individual tests. So I will definitely think twice before putting something in those next time!
3. How can or should tests access internal functionality?
In a modular code base, different parts will communicate with each other using their specified public APIs. Each module hides its dirty parts from the others so they do not have to deal with it. Unit tests can then call these API methods and see if they do what is expected. That also shows that modules should be really small, because anything hidden inside a module will not be tested directly.