Configuring BizTalk 2020 Application Insights telemetry behind a firewall

BizTalk 2020 allows you to connect it to an Application Insights instance, where it will send tracking data as customEvents so you see what's going on. However, getting it working in an environment where security is important and the network team want to open the fewest paths through the firewall as possible is an exercise in patience. This blog post is as much aide memoire for me as an information post for you.

Annoying GUIs 101

The first issue you'll come across is the dialog for connecting BizTalk to Application Insights. It has a text box, so you expect to be able to simple paste your InstrumentationKey in there and call it done, right? Wrong! This part of the documentation isn't kidding when it makes you step through a whole bunch of stages, and that's where the first issues kick in.

I suppose it makes sense - you can either configured telemetry into Application Insights or into Event Hubs, and the GUI lets you do both. In the background it puts the config into an SSO Application (which you can't see or edit), so why bother with the useless text box?

The problem here is that you need to authenticate against Azure AD, then it lists your subscriptions, and then the resource groups, and finally the Application Insights instance. That means talking to both the Azure AD authentication services and the Azure Admin API. The good news is that you can turn that access off once configured.

Whither goes my data?

The second issue is that BizTalk doesn't appear to find the appropriate ingestion endpoint for the App Insights instance. This is the part that really consumed my time. I'm assuming that it reads the IngestionKey from the instance in Azure but not the connection string. In our customer's environment we had AppInsights deployed to UK South and opened firewall ports for the address range in the UK (more on that in a bit) but no data flowed. After a while poking around with their team and their hardware edge device, I decided to build a lab so I could figure out what was happening.

To do that, I downloaded pfSense and create a virtual machine that could sit between my BizTalk test rig and the internet. That allowed me to replicate the rules my customer had already created, monitor the traffic flowing from BizTalk and find the correct (and smallest) list of Azure services and addresses I needed to open access for.

To figure out the rules we needed, our initial work had begun with the Azure Monitor IP Address documentation. The networks team at my customer told me their device automatically updated address blocks from the Azure json file that you can find referenced on that page. I needed to download the file in order to create my firewall rules.

To cut to the chase, the end result of my investigations shows that BizTalk is sending it's telemetry to North Europe, even though the AppInsights instance is in UK South. To find that, I began by including the UK South monitor addresses in my rule. When that failed I went big and included all the monitor addresses globally, which worked. Checking the connections showed me that it was talking to an address in the North Europe range, so I could then restrict to just that address range.

What does this mean

If you are in a regulated environment or your customer is keen on security and restricting where data flows, this is a Big Thing. For my customer, having data flowing out to Dublin then back across the Azure network to UK South is acceptable, but it limits what we can send. For some customers this kind of behaviour would be a show-stopper.

Ultimately, the BizTalk team need to update their code to use the cConnection string for the Application Insights instance. That clearly denotes the ingestion endpoint, which is most definitely in the same region as the instance. With the best will in the world, however, this can only be done within their update cycle so it's unlikely to be quick (yes, I have passed all my investigations and findings to the team and in fairness to them they use the AppInsights SDK and followed the recommended practices in place when they wrote the code).