Manage Code Coverage by Unit Tests with Sonar
Code coverage by unit tests is one of the practices that we are pushing the most at SonarSource. We have currently more than 10,000 unit tests running daily to cover the platform and keep adding every day. Of course, Sonar is not the place to start from to do Test Driven Development, Sonar is not an IDE, Sonar will not create unit tests for you and Sonar will not replace precious Agile developers. But once the first bunch of code has been commited, you should certainly use it to manage unit tests and code coverage. The objective of this post is to show how to use Sonar to meet this requirement.
The first thing you can do with Sonar is to monitor the overall coverage of an application. For example, in our team we have a requirement that overall coverage for Sonar should be around 70% which we can follow in the dashboard. In the same widget, you will also find information about unit tests execution, line coverage and branch coverage to complete the coverage indicator:
As for all other metrics, the next thing you want to do after getting a measure is to look at the evolution of this measure throughout time and answer the following questions: where are we coming from, what is the trend? Indeed, it makes a big difference whether coverage value was 70%, 90% or 30% 6 months ago and should lead to different actions. In our case, it has been fairly consistent during the past 3 years:
Now that we have information about overall value and its evolution, can we consider that we have sufficient information to manage the code coverage of an application? Surely not. We also need to get an idea of the coverage of underlying components to find out about distribution of coverage and pick areas that are badly covered. This is why Sonar provides a treemap component to be able to break down by underlying component:
On this one, Sonar::Core seems to be in a much better shape than Sonar:Batch.
With this information, we get a much better understanding at a high level but as for every measure which is an average, it always has the same drawback that it can hide very different situations: are all your files covered at 70% or are many of them covered at 100% and the rest at 0%? To find out the real situation of coverage, you can use the drilldown by clicking on the measures you see in the dashboard. This will provide you with 2 new information:
Either you click on overall figure and it will show you the classes that have the lowest coverage. This enables you to track the requirement by file rather than by application:
Or you click on the line coverage and it will show you the files that have the most uncovered lines and therefore where the biggest effort should be made:
Once you start monitoring the coverage by file (on Sonar, we have set a requirement of 65% per file), the SQALE Technical Debt plugin (commercial product) will provide information about the size of the effort to the objective in days:
You can also get a view that combines both information to highlight risky areas by using the cloud service. Size of words shows the size of classes and their color the coverage. In this service, you will be hunting for big red classes!
All this is very good and pretty straight forward when working on a brand new application or one which has a decent coverage, but this is obviously not the majority. To manage coverage of legacy applications with Sonar, you can use the differential mode of dashboards to check how you are doing against a pre-defined baseline. This will provide you some information about the general trend (here our code coverage has increased by 0.6% and both line and branch coverages have improved). On the top of it, it will show the coverage on new and updated code. This is pretty good as it enables to set the same type of requirements that you have for new applications and manage them since a date. In our case here we see 78.8% which is very good
Now that you have had a good overview of how Sonar can help manage your code coverage, I guess that you must be eager to start using it… therefore here are pointers onto how to compute code coverage information into Sonar.
Basically, there are two ways to feed Sonar with this information:
- Reusing existing reports that have previously been generated
- Launching execution of unit tests directly in Sonar
Once you are done, you can also consider to start looking at code coverage by integration tests.