Code coverage statistics are very useful. They tell you how much of your code never gets executed during the unit tests. So you always know what to write unit tests for, because 100% coverage doesn’t exist. And sometimes you can’t cover all code. That’s why normally you are happy, if the coverage is above a certain thresh hold, but having a higher coverage is always ok
One of the typical coverage tools for CommonJS based code is istanbul. Sadly it doesn’t work just out of the box, because the coverage calculations and tests aren’t run in the same JS engine. The tests run in Firefox and the calculations happen outside in NodeJS.
By default Istanbul uses a global variable to store the coverage information. The problem begins with the CommonJS environment of the Add-on SDK not having a global environment. To work around that, I’ve made a module that exports an object which istanbul then treats as its global object. But to do that, the Instrumenter, the thing that makes your code inspectable during unit tests, has to be modified to use that module instead of the Node global. Luckily this is just a modification of one line in a function in the prototype of the Instrumenter. This happens in the main module of the istanbul-jpm package, which I created.
/* The second item in the array differs from the normal
* Instrumenter, everything else is the same. */
code = [
"%STRICT%",
"var %VAR% = require('istanbul-jpm/global').global;",
"if (!%VAR%.%GLOBAL%) { %VAR%.%GLOBAL% = {}; }",
"%VAR% = %VAR%.%GLOBAL%;",
"if (!(%VAR%['%FILE%'])) {",
" %VAR%['%FILE%'] = %OBJECT%;",
"}",
"%VAR% = %VAR%['%FILE%'];"
]
Having a custom module providing the “global” scope also makes bridging the gap between Firefox and Node really easy. It just serializes the “global” object when the extension is unloaded (after all unit tests are done) and writes it to disk. Funnily enough the module is called global. There is a module for node that then reads and parses that file called global-node.
/* Loading the collected coverage data into the node
* environment. */
global.__coverage__ = require("istanbul-jpm/global-node").global.__coverage__;
Only the global.js from the istanbul-jpm package is needed during unit tests, everything else can safely be .jpmignored. Sadly you have to do that yourself, since JPM doesn’t respect the .jpmignore files from extensions.
And that’s all the extra magic needed. The rest is magic that needs to happen whenever you collect coverage data. But because generating coverage statistics is quite complicated I’ll explain the other steps too:
Before the unit tests are ran, the Instrumenter adds the tracking logic to your modules (normally everything in your lib folder and maybe in your root). I haven’t bothered to figure out a way to do coverage analysis for content scripts, by the way.
But for coverage data to be collected the modules generated by the Instrumenter have to be loaded in the unit tests and not the normal modules. Common practice is to create a require-helper module, that checks an environment variable and either loads the module from its actual location or redirects it to the instrumented version.
An example implementation for instrumented modules being in the /coverage/instrument/lib/ folder, the following code does exactly what is described in the previous paragraph. If the environment variable JPM_MEASURING_COVERAGE is set, it redirects to the instrumented versions.
const system = require("sdk/system");
module.exports = function (path) {
if(system.env.JPM_MEASURING_COVERAGE)
return require('../coverage/instrument/test/' + path);
else
return require(path);
};
After the unit tests a report has to be generated. This can have many forms, and really depends on what you want to do with coverage statistics.
I use grunt to manage all of this, a bootsrapped Gruntfile.js can be found in the README,md of the istanbul-jpm module. There is also the monstrosity, that is the Gruntfile.js of my jtvn extension.
The istanbul-jpm module helps close the gap between the normal Node environment and JPM tests for coverage analysis with istanbul. It’s pretty young, but there’s not much that can go wrong. Give it a try!