Siesta 5.1.0 released

Today we are glad to announce the Siesta 5.1.0 release.  In this release, we present a completely new, re-engineered code coverage module. It might seems strange to release 5.1 immediately after 5.0, but this is what the semantic versioning rules dictate, so we follow.

Code coverage in previous Siesta versions

When we built the prior version of the code coverage module, we made a design choice which unfortunately lead to its stagnation. We decided that the code coverage generation report should be available through the Siesta web interface. It seemed like a good idea – to instantly see the coverage results update live as you are writing your tests. But, with this approach, the code instrumentation had to happen in the browser and Istanbul (the code coverage library we use) works only in Node.js. We also added support for the “coverage units”, which was basically instrumenting parts of a big “bundled” JavaScript file separately.

To achieve the goals mentioned above, we had to heavily patch Istanbul. This of course made it very hard to upgrade Istanbul to newer versions. We also had to patch the Istanbul reports – so we ended up supporting only some of them. Because of this, Siesta’s code coverage module got stuck, it had some bugs which were hard to fix and it was not possible for us to adopt the new ES6 syntax, like arrow functions which is now supported by all modern browsers.

Since version 5, Siesta also started to support testing Node.js code, and code instrumentation in the browser just was not general enough to support both browsers and Node. So we decided that this module should be rewritten from scratch.

The new code coverage in Siesta 5.1.0

With the new code coverage module we just provide a thin integration layer on top of Istanbul and we do not patch its code. Our goals with the re-implementation were:

• Easy upgrades to new versions of Istanbul to adopt new features
• Support both in-browser JavaScript and Node.js
• Ability to generate all reports supported by Istanbul.

To achieve these goals, the code instrumentation and reports generation have to happen in Node. As a result, code coverage is now available only from the command line and not in the web interface and we found this to be a reasonable trade-off.

The code coverage in Siesta 5.1.0 is controlled by the --nyc.* command line option group. Such options, like --nyc.include or --nyc.reporter are translated directly to the nyc (Istanbul’s command line interface tool), so they work in the exact the same way as in nyc.

To collect code coverage information and generate a report, you should specify the --nyc.reporter option (old name --coverage-report-format is still supported as an alias). All Istanbul reports listed here are supported.

For example, to get a text coverage report for Node.js code, simply run:

> bin/nodejs examples/nodejs/code-coverage --nyc.reporter=text

Or, to generate an HTML coverage report for browser JS code:

> bin/webdriver examples/sencha-extjs --nyc.include='**/5.code_coverage' --nyc.reporter=html --include code_coverage

The code coverage module works with cloud testing providers like Sauce Labs too. The only detail is that the Sauce Labs tunnel should be configured with additional options, so one should allow the Siesta launcher to establish it:

> bin/webdriver examples/sencha-extjs --nyc.include='**/5.code_coverage' --nyc.reporter=html --include code_coverage --saucelabs $SL_USER,$SL_KEY,$SL_DOMAIN

Code coverage for browser JavaScript code

As you can imagine, testing browser JavaScript code is quite different from testing Node.js code. To support code coverage in the browser and avoid patching Istanbul, we have implemented a special “semi-transparent” proxy server. It just passes through all requests, except those for JavaScript and matching --nyc.include and --nyc.exclude options. This proxy is then automatically set for all WebDriver sessions.

For the purposes of including/excluding, the URL of the request is transformed into a “virtual” file name. For example, the URL https://project.com/lib/app.js?cache=123 becomes /http/project.com/lib/app.js Then the content of the request is passed to Istanbul for instrumentation. We use Istanbul’s caching, meaning any repeating requests for the same file will be processed much faster (it will still take some time, since Istanbul calculates the hash of the file content).

When testing over https, which is becoming more and more popular, Siesta automatically generates self-signed certificates for all encountered hosts. IE11 may require additional setup because of this since its WebDriver version does not support the “acceptSslCerts” capability. Please see code coverage guide for more information.

One caveat when instrumenting browser code is that by default *all* JavaScript code is instrumented. This may cause a heavy CPU load if your project uses a lot of 3rd party libraries for example. In such case we recommend that you to explicitly specify what code should be instrumented with the --nyc.include option.

Code pre-instrumentation

With Siesta 5.1.0, it is now possible to pre-instrument the codebase. This may be needed if you want to bundle several JavaScript files into one and then get coverage results for individual files (e.g using WebPack). For a pre-instrumented codebase, you can disable the on-the-fly instrumentation with the --nyc.instrument=false option.

Pre-instrumentation can be done using:

• babel-plugin-istanbul Please find more information in the nyc documentation
• The instrument command of the nyc tool. More information can be found by running npx nyc instrument --help (or node node_modules/.bin/nyc instrument --help)

API changes

So what are the changes in the API? Let’s review the configuration options from the previous implementation and their equivalents in the new:

• enableCodeCoverage – project option, removed. Code coverage is now always controlled from the command line with the --nyc.reporter
• coverageUnit– project option, removed. There’s no “coverage unit” concept in the new implementation. It was supporting getting better reports from the bundled code. Same result can be achieved with the pre-instrumentation of the source code before bundling.
• includeCoverageUnit, excludeCoverageUnit– project options, removed. Use --nyc.include, --nyc.exclude instead.
• --previous-coverage-report command line option has been removed. To create a combined coverage report from several consequent test suite launches, specify the --nyc.clean=false for the 2nd and following launches

Integration with your CI dashboard

At Bryntum we use TeamCity as our Continous Integration tool and we now have our code coverage reports displayed with each build. This makes it easy to keep an eye on our coverage levels to make sure we don’t release under tested code. Below you can see our report for the Bryntum Grid inside our TeamCity dashboard:

Summing up

Siesta 5.1.0 presents a new code coverage module that fully supports modern JavaScript. Additionally it is capable to covering browser code too. We encourage you to give it a try and as always, your feedback is very welcome in the comments section or our forums. Happy coverage testing!

Download Siesta 5.1.0 in the customer zone