Set up a CI/CD pipeline with unit testing and code coverage for your Team Services extension

We receive a notification that there’s a pull request to review for our Team Services extensibility project. We discuss the proposed changes as a team, eventually merge the new feature(s) into the master branch, and close the pull request. Done … right?

Not quite. The merge triggers our automated CI/CD pipeline, which consistently builds, tests and delivers our package to the Visual Studio marketplace.

This post should be a valuable reference, if you’re interested in how we configured our Countdown Widget pipeline and how the team included testing and code coverage. Let’s walk through the configuration of our pipeline.

Build

We start with the continuous integration build, made up of thirteen steps.

The magic around unit testing and code coverage happens during steps 8-11. See Testing a Team Services extension and tracking code coverage for more details.

#	BUILD STEP	BUILD SNIPPET	WHAT’S HAPPENING?
1	Whitesource		Scan and detect security vulnerabilities, problematic OSS licenses and quality issues. See Manage your open source usage and security in your pipeline for details. You’ll notice that we’re still experimenting with this task and its location in the build. Here we scan before the build, while in Set up a CI/CD pipeline for your Team Services extension we scan after the build. Watch our Whitesource tag for updates on our findings.
2	Security “stuff”		We’re experimenting with a collection of internal security tools to automate tasks like checking for sensitive term violations as part of the pipeline.
3	NuGet restore		Restore NuGet packages as configured in the CountdownWidget.sln solution.
4	npm install		Install the NPM package dependencies as defined in package.json.
5	grunt prepare for build		Perform the copy:main (copy) custom task defined in gruntfile.js copying seven files.
6	Replace tokens		We replace the INSTRUMENTATION key in the TelemetryClient.ts file with the value of the variable with the same name. Token Regex is “__(w+)__”, which means that we’re looking for one or more letter, digit or underscore characters, surrounded by a double underscore “__name__”. Variables INSTRUMENTATIONKEY defines the Application Insights instrumentation key. We’re using a variable to ensure we do not leak the instrumentation key into the OSS repo. system.debug can be set to true for detailed and verbose logging during the build.
7	Build solution		Build solution as defined in CountdownWidget.sln.
8	Test Assemblies		Run tests with Visual Studio test runner looking for TypeScript test files named *Spec.ts. The Chutzpah package, containing Jasmine and its test adapter, were installed to the packages folder as part of step 3 – NuGet restore.  We are currently running 18 Jasmine based unit tests, as defined in CountdownCalculatorSpec.ts.
9	Run python Icov tp cobertura script	Arguments: $(Build.SourcesDirectory)CountdownWidgetlcov_cobertura.py  $(Build.SourcesDirectory)CountdownWidgetCountdownWidgetlcov.dat –output $(Build.BinariesDirectory)Cobertura.xml	Convert the file lcov.dat line coverage output to Cobertura-compatible XML format using the lcov_cobertura.py python script.
10	Run Report Generator	Tool: $(Build.SourcesDirectory)/ CountdownWidget/packages/ ReportGenerator.2.5.1/tools/ ReportGenerator.exe Arguments: -reports:$(Build.BinariesDirectory)Cobertura.xml -targetdir:$(Build.BinariesDirectory)reports -reporttypes:Html;Xml;XmlSummary	Convert the Cobertura.xml file to HTML format, using the ReportGenerator we installed to the packages folder as part of step 3 – NuGet restore.
11	Publish Code Coverage Results	TEST RESULTS: CODE COVERAGE DETAILS:	Publish the code coverage results from the formatted Cobertura.xml file.
12	Package Extension		We package the extension using the VSTS Developer Tools Build Tasks extension and generate the output.vsix artifact. Extension ID is appended with Alpha to differentiate it from deployed DEV, BETA, and PROD packages.
13	Publish Artifact		We publish the artifact, output.vsix, to $(Extension.OutputPath) to be picked up by the release.

Release

Next we continue with continuous delivery (CD), made up of three environments. With the exception of configuration values, this part of the pipeline is identical (consistent) to the one we covered in up Set up a CI/CD pipeline for your Team Services extension.

Continuous Delivery is the ability to use the output from the CI to build and deploy the new known good build to one or more environments automatically (bit.ly/28PWsfk). There is a subtle difference between Continuous Delivery and Continuous Deployment. The latter is to a single environment. A small team might only implement Continuous Deployment because each change goes directly to production. Continuous Delivery is moving code through several environments, ultimately ending up in production, which may include automated UI, load and performance tests and approvals along the way. – Extract from DevOps – Applying DevOps to a Software Development Project

DEV

The environment with one task is triggered after a successful release creation. Both the pre-deployment and post-deployment approvals are automatic, ensuring that the development team is always working with the latest version. There’s no manual intervention.

#	RELEASE STEP	BUILD SNIPPET	WHAT’S HAPPENING?
1	Publish Extension		We publish the extension to the alm-rangers publisher, using the VSTS Developer Tools Build Tasks extension.  The published private extension is shared with our BETA sandbox-one and used for exploratory and user acceptance testing. The extension ID is suffixed with BETA to make it simple to identify this extension.

BETA

The environment with only one task is triggered after a successful deployment to the DEV environment. The BETA approvals mandate that one of the specified users approve the release, to ensure that a high quality bar can be met. Users typically include the project lead and program manager.

#	RELEASE STEP	BUILD SNIPPET	WHAT’S HAPPENING?
1	Publish Extension		We publish the extension to the alm-rangers publisher, using the VSTS Developer Tools Build Tasks extension.  The published private extension is shared with our BETA sandbox-one and used for exploratory and user acceptance testing. The extension ID is suffixed with BETA to make it simple to identify this extension.

PROD

The environment with one task is triggered after a successful deployment to the BETA environment. The PROD approvals mandate that all of the specified users approve the release, to validate that the aka.ms/vsarDoD (definition of done) is met. Users typically include the project lead, program manager, and product owner.

#	BUILD STEP	BUILD SNIPPET	WHAT’S HAPPENING?
1	Publish Extension		We publish the extension to the ms-devlabs publisher, using the VSTS Developer Tools Build Tasks extension.   The published extension is public and available on the marketplace.

That’s it!

Now that you have an understanding of how our pipeline works, you should explore how you can configure your continuous integration and delivery pipeline by #RubDevOpsOnIt.

Other pipelines examples

Reference Information

DevOps – Applying DevOps to a Software Development Project, lcov-to-cobertura-xml, Set up a CI/CD pipeline for your Team Services extension, Testing a Team Services extension and tracking code coverage, VSTS Developer Tools Build Tasks