Errors with TMG + Exchange Edge Connector + FPE resulting in rejected emails

Frustrating errors with little or no explanation…

Once again I find myself wiring a blog post in order to save people the time we spent figuring out what was going on and getting help with our fault.

We have Microsoft Threat Management Gateway installed at our perimeter. The Exchange 2010 Edge Connector is also installed as our mail gateway, and finally Forefront Protection for Exchange (FPE) deals with mail scanning. There are some gotchas that can trip you up when installing that lot, which I suppose I should put in another post, but the overall result is that you can manage the email filters (spam, malware etc) from within TMG’s console. It’s a nice, unified management interface and a good solution. When it works…

Our problem, when it occurred, was simple in its symptom – emails were being rejected by our email server. Emails were rejected that we didn’t think should be, and we fiddled with IP allow list settings in TMG for a while to no avail.

Looking in the event logs we noticed that there were lots of messages with the event ID 31506 and description:
Forefront TMG detected changes in Microsoft Exchange Server or Microsoft Forefront Protection configuration, and reapplied the e-mail policy configuration on server .

There are quite a few people on the net with this error. One of the things they experienced, and we also saw, was that the Microsoft Forefront TMG Managed Control service was not running. Lots of messages suggested that restarting that service should do the trick. Lots of other messages also said that the service wouldn’t restart. We saw that too.

Clearing the IP block list is supposed to fix that fault. At first it did for us. To do this open the Exchange Management Shell on your TMG server and use the following powershell:

get-ipblocklistentry will return a list, if there are any entries.

remove-ipblocklistentry is the command to delete an item from the list, so get-ipblocklistentry | removeipblocklistentry –confirm:$false will get the block list, pipe the items through the remove command and delete them. The –confirm:$false tells the command just to get on with it and not prompt for confirmation.

The trouble was, whilst this worked, the solution was only temporary. We then tried something else (and promptly forgot about it, which will become important in a while).

We now found ourselves in a situation where the Microsoft Forefront TMG Managed Control service would not start. This time, whenever we tried to start the service we saw errors in the log with event id 31308 and description:
The Forefront TMG Managed Control service failed to initialize. Error information: Value does not fall within the expected range.

Nothing we could do would get this going. Again, people on the web suggested clearing the IP block list but doing so made no difference. We also saw the block list growing rapidly and incoming emails bounced off the spam filter, which we couldn’t disable because the TMG managed control service wouldn’t run.

We knocked up a quick powershell script to keep the wolves from the door and dug further. For those in the same boat, the following script simply clears the contents of the block list every 30 minutes and then sleeps.

for (;;) { write-host Clearing block list get-ipblocklistentry | remove-ipblocklistentry -Confirm:$False write-host Cleared block list. Now waiting 30 minutes... start-sleep -seconds 1800 }

At this point I found a very obscure post talking about the original error we saw and that there is an ‘unofficial’ hotfix available via support. I’ve encountered these before and you have to log a support call to get them, because Microsoft want to make very sure that you are having the problem addressed by the fix. Usually these things get rolled into updates at a later date, once they’ve been through the full raft of testing needed to do so.

The hotfix is TMG-ISASE38387-UNOFFICIAL-AMD64-GLB and it was given to us by Microsoft Support in .exe and .msp flavours. Before you all get in touch, no I won’t give you the files. There’s a reason that the product support guys want you to log a call to get hold of this patch and not just install it randomly.

I dutifully installed the hotfix. It made no difference (or so it appeared) – the Microsoft Forefront TMG Managed Control service still refused to start.

It was at this point that we had an epiphany. In our attempts to stop email bouncing we had added items to the IP allow list, but done so through the Forefont Protection for Exchange management console. Interestingly (and importantly), the entries we had added here were not appearing in the Exchange IP Allow list (found by using get-ipallowlistentry in the Management Shell). We removed these entries and tried again.

The service started!

We then configured the IP allow list in TMG and applied the configuration changes. This time, checking the IP allow list via the management shell returned the entries we were expecting. Our email system has been functioning correctly ever since. The hotfix resolved our original error and our realisation of our mistake corrected the second problem.

Important Points

• The error, event id 31506, description “Forefront TMG detected changes in Microsoft Exchange Server or Microsoft Forefront Protection configuration, and reapplied the e-mail policy configuration on server .” is a known fault.
• An unofficial hotfix exists, TMG-ISASE38387-UNOFFICIAL-AMD64-GLB, which addresses the issue, but you MUST engage with Product Support to get it.
• I am told that the fully tested fix will be included in an update rollup for TMG which is due in February.
• Don’t make configuration changes via the Forefont Protection for Exchange Server Console if your are running in this integrated setup. As we found, entries to things like the IP Allow List (and I’m guessing IP Block List) will cause the Microsoft Forefront TMG Managed Control service to fail to start with an error, event id 31308 and description ”The Forefront TMG Managed Control service failed to initialize. Error information: Value does not fall within the expected range.”
  Note: We also found that entries added to the IP allow list in FPE didn’t get listed when using the get-ipallowlistentry PowerShell command in the Exchange Management Shell.
• If your Microsoft Forefront TMG Managed Control service does fail to start, check that you haven’t made configuration changes through FPE. The Exchange Management Console on TMG is clever enough to not let you make changes like this, but FPE isn’t.