The Problem

Due to recent changes at NIST and the way they host the NVD DB the dependency-check/DependencyCheck tool that underpins the Azure DevOps OWASP Dependency Check is struggling to download the vulnerabilities DB.

This is all discussed in this support thread. The root cause of the problems is a mixture of many updates to the NVD DB, increasing the download size, and tighter throttling of the API.

The net-effect of this is that download the NVD DB now takes hours, not minutes, with many timeouts and retries.

So, making the use of some from of local caching vital.

The Analysis

Up to now I have been using scheduled maintenance pipelines to do my NVD DB updates.

However, these have an architectural problem, each of my self hosted build agents performed their own daily NVD DB download. The problem with this, other than the waste of bandwidth, is that they used the same API key, so have the effect of throttling each other, it was random luck as to which completed and which timed out.

I needed a revised plan, where I download the DB once, then somehow synchronized the download DB to all my other self hosted agent. It needed to:

• Minimize my API calls
• Avoid using a shared DB as we aim to keep our self hosted build agents as close to the MS hosted ones as possible

The Solution

My solution was a modified version of existing scheduled maintenance process

• Within an Azure DevOps Pipeline that runs on a daily schedule
  • On a single self hosted agent updated it’s copy of the NVD DB.

    If the DB is already fairly up to date this is a quick process.

  • Publish the NVD DB as a pipeline artifact
  • Using a PowerShell script find all the Agent Pools and Self-hosted Agents (other than the one that has just been updated) and trigger a pipeline run on each agent of a new pipeline that pulls downloads the current build’s NVD DB artifact and copies the files to the correct location on the agent.

You can find the outline of the process in this gist

So a slightly convoluted process, but it has a low impact on the NIST API, it is quick to run and does not require me to grant special permissions to any of my self-hosted agents to see a shared DB.

For the original version of this post see Richard Fennell's personal blog at Lower API impact way of updating NVD Vulnerability Dependency data