This is how you can automate Coverity Scan using Travis CI – especially if you have a complex build matrix:
- create an additional matrix entry you will exclusively use for submission to Coverity
- make sure you use your regular git master branch for the scans (so you can be sure you scan the real thing!)
- schedule a Travis CI cron job (daily, if permitted by project size and Coverity submission allowance)
- In that cron job, on the dedicated Coverity matrix entry:
- cache Coverity’s analysis tools on your own web site, download them from there during Travis CI VM preparation (Coverity doesn’t like too-frequent downloads)
- prepare your project for compilation as usual (autoreconf, configure, …) – ensure that you build all source units, as you want a full scan
- run the cov-int tool according to Coverity instructions
- tar the build result
- use the “manual” wget upload capability (doc on Coverity submission web form); make sure you use a secure Travis environment variable for your Coverity token
- you will receive scan results via email as usual – if you like, automate email-to-issue creation for newly found defects
Why so complicated? After all, there is a Coverity addon for Travis CI. That’s right, and it probably works great for simple build matrices. Rsyslog uses Travis quite intensely. At the time of this writing, we spawn 8 VMs for different tests and test environments plus the one for cron jobs (so 9 in total). With the Coverity addon, this would result in 9 submissions per run, because the addon runs once per VM requested. Given that rsyslog currently has an allowance of 3 scans per day, a single run would use up all of our allowance plus 6 more which would not succeed. Even if it would work, it would be a big waste of Coverity resources and be pretty impolitely both to Coverity and fellow users of the free scan service.
With the above method, we have full control over what we submit to Coverity and how we do it.With our current scan allowance, we use exactly one scan per day, which leaves two left for cases where it is useful to run interim scans during development.
Why do you use master branch for the scan? Coverity recommends a dedicated branch.
In my opinion, software quality assurance works well only if it is automated and is applied to what actually is being “delivered”. In that sense, real CI with one Coverity run per pull request (before we merge!) would be best. We can’t do this due to Coverity allowance limitations. The next best thing is to do it as soon as possible after merge (daily in our case) and make sure it is actually run against the then-current state. Introducing a different branch just for scan purposes sounds counter-productive:
- you either need to update it to full master branch immediately before the scan, in which case you can use master directly
- or you do update the “Coverity branch” only on “special events” (or even manually), which would be counter the idea to have things checked as early as possible. Why would you withhold scanning something that you merged to master, aka “this is good to go”? Sounds like an approach to self-cheating to me…