Another new problem when generating build agents using Packer

The Problem

I have been using Packer to generate our Azure DevOps Build agent disk images for a while now. The advantage of this process is that our Azure DevOps self-hosted agents have the same installed software, and hence capabilities, as the Microsoft hosted ones. So it is trival to move a build pipeline from self-host agents to either Microsoft hosted agents or Managed DevOps Pools.

However, the generation process for these agent VM images has been the gift that keeps giving and has generated a few blog posts …

• Changes in how PATs need to be passed into Packer
• The change from using VHD to IMAGE generation in Packer

So it was not unexpected that when I came to rebuild our agents this time I found a new issue.

The VM images generated are now in HyperV GEN2 format, not GEN1, due to changes in the updated Packer .pkr.hcl templates files.

This caused me to have to make some changes.

The Changes

Azure Compute Gallery

The first change was that I needed to delete the old GEN1 VM Images Definitions in my Azure Compute Gallery. I had to delete both the previously provisioned image copies (think files) and the actual definitions (think folder). This change was required as a definition can contain only either GEN1 or GEN2 images, not a mixture.

For us, this deletion was not an issue, as the images are only placed into the Azure Compute Gallery as a temporary location so I can download them.

Download Process

I have a script that uses New-AzDisk to export the new disk image from the gallery and download it. The issue is that this command only natively supports VHD format, which is a problem for Lability, the tool we use to manage the creation of the VMs in Hyper-V, as it needs VHDX format for GEN2 VMs.

This means I have had to alter my export and download script to do a VHD > VHDX conversion after the download

The complete process is now as follows:

1. [Done once] Create an Azure Compute Gallery instance in your subscription.

2. Run Packer to generate your generalised VM Image

3. In the Azure Portal view the newly created VM Image and select ‘Clone to a VM Image’

   • Select the previously created Azure Compute Gallery
   • Provide a version number, we are using one based on the OS of the images e.g. 2025.0.1
   • If it is the first time you are cloning a VM image, create a new ‘Target VM Image definition’ with a suitable name, for all subsequent clones just select the existing definition target
   • Pick the replication rules that meet your needs, I used local replication on premium storage.

   The cloning of the image version takes around 30 minutes for the 250Gb image.

   If you don’t want to use the portal, you could use the Azure CLI, using the command az sig image-version commands i.e

   az sig image-version create --gallery-name MyPackerGallery --resource-group myrg --gallery-image-definition BuildAgent2022 --gallery-image-version 2.0.1 --managed-image /subscriptions/<GUID>/resourceGroups/BMPACKER/providers/Microsoft.Compute/images/buildagent --target-regions westeurope=1=premium_lrs --location westeurope
   

   You need the --location else your subscriptions default location is used, which may not be where your VM Image is, resulting in the somewhat confusing error given you have provided a full URL for the image and your source and target are in the same region and resource group.

   (InvalidParameter) Gallery image version publishing profile regions 'westeurope' must contain the location of image version 'North Europe'.
   

4. Once the replication has completed, consider deleting the VM Image that was created by Packer as it is no longer needed

5. You can then download the generalised VHD and convert it to VHDX using PowerShell.

param (
    $subscription  # "My Subscription",
    $rgName # "packer",
    $galleryName  # "PackerGallery",
    $galleryDefintionName  # "BuildAgent2025",
    $galleryImageVersion # "2025.0.1",
    $targetDir # "c:\download",
    $generation # 'GEN2'
)
write-host "This script uses the Az PowerShell module"
write-host "    Install-Module -Name Az -Repository" write-host "It also assumed that AZCOPY.EXE is in the current folder`n`n`" 

write-host "Connect to Azure subscription '$subscription'" 
Connect-AzAccount -Subscription $subscription

write-host "Connecting to Azure subscript '$subscription'"
select-AzSubscription $subscription
$imgver = Get-AzGalleryImageVersion -ResourceGroupName $rgName -GalleryName $galleryName  -GalleryImageDefinitionName $galleryDefintionName -Name $galleryImageVersion

write-host "Downloading VHD for $galleryDefintionName $galleryImageVersion"
$imgver
$galleryImageVersionID = $imgver.Id

write-host "Creating temporary disk"
$diskName = "tmpOSDisk"
$imageOSDisk = @{Id = $galleryImageVersionID}
$OSDiskConfig = New-AzDiskConfig -Location $imgver.location -CreateOption "FromImage" -GalleryImageReference $imageOSDisk
$osd = New-AzDisk -ResourceGroupName $rgName -DiskName $diskName -Disk $OSDiskConfig

$downloadPath = $targetDir + "\" + $galleryImageVersion + ".vhd"
write-host "Granting access to temporary disk"
$sas = Grant-AzDiskAccess -ResourceGroupName $rgName -DiskName $osd.Name -Access "Read" -DurationInSecond 18000

# We need to up the timeout else only get 35Gb of the 250Gb VHD
write-host "Downloading VHD to $downloadpath - this will take some time"
.\azcopy cp $sas.AccessSAS $downloadPath

if ($generation -eq "Gen2") {
    write-host "Convert VHD to VHDX - this will take some time"
    $vhdpath = $downloadPath.replace(".vhd",".vhdx")
    Convert-VHD -Path $downloadPath -DestinationPath $vhdpath -VHDType Dynamic
} else {
    $vhdpath = $downloadPath
}

Lability

The final change required was to update my media registrations in Lability to add the Customdata block to specify the image generation, as opposed to letting it default to GEN1, and yes before you ask the MediaType should be VHD even though the file is a VHDX!

@{Images =
# OS
@{
	Id           = 'Build_Agent_VS_2025'
	Filename     = '2025.0.1.vhdx'
	Architecture = 'x64'
	Checksum     = 'CB234F153868C45233589B5FF92362F'
	MediaType    = 'VHD'
	Description  = 'Azure DevOps 2025 Build Agent'
	Uri          = '\\myshare.domain.com\Images\Azure DevOps Agent\2025.0.1.vhdx'
	CustomData   = @{
		Generation = '2'
	}
}
}

Once the VHDX was on our local network, and the media registration correct, I was able to build my self hosted agents as normal using Lability on Hyper-V

Conclusion

So, as seen previously, the changes required in my process due to ‘upstream changes’ were not as bad as first feared. However, they have added a few more hours to the export/download process. Though, this is not a major issue as it is a process done once for each image, and something I was already running as an overnight process.

For the original version of this post see Richard Fennell's personal blog at Another new problem when generating build agents using Packer