Web Application Proxy Failure Following Outage

Following a ‘hiccup’, involving a Web Application Proxy (WAP) server, internal services were no longer being published to the outside world.

After some investigation, both the ADFS and WAP services showed as stopped on the server. Attempting to start the ADFS service from the services console produced the following error:

Windows could not start the Active Directory Federation Service service on Local Computer.
Error 1064: An exception occurred in the service when handling the control request.

Under the System section of the Windows Event Log, the following error was shown:

Event ID: 7023
The Active Directory Federation Services service terminated with the following error:
An exception occurred in the service when handling the control request.

Followed a few moments later by the following error:

Event ID: 7023
The Web Application Proxy Service terminated with the following error:
A certificate is required to complete client authentication

Looking in the ‘AD FS’ section of the Event Log (under ‘Applications and Services Logs’), the following errors were shown (note that the first error was generally shown multiple times, followed by a single instance of the second error):

Event ID: 383
The Web request failed because the web.config is malformed.
User Action:
Fix the malformed data in the web.config file.
Exception details:
Root element is missing (C:WindowsADFSConfigmicrosoft.identityServer.proxyservice.exe.config)
Root element is missing.

Followed by:

Event ID: 199
The federation server proxy could not be started.
Reason: Error retrieving proxy configuration from the Federation Service.
Additional Data
Exception details:
An error occurred when attempting to load the proxy configuration.

Checking the file at C:WindowsADFSConfigmicrosoft.identityServer.proxyservice.exe.config showed that while the file size was still indicated as 2k, the file was blank.

I’ve seen a number of reports online indicating that WAP seems happy to chew up the contents of this configuration file following an outage, although I can find no information on why this might happen. If you have a backup of the file in question, it should be a simple matter to restore this file and restart the ADFS and WAP services to restore service. If you don’t, and have no other example server from which you can pull a similar copy of the file then the following steps must be taken:

  1. Remove the Web Application Proxy role from the server. Once this is complete, a reboot will be required.
  2. Re-add the Web Application Proxy role to the server.
  3. Once this is complete, initiate the configuration wizard.
  4. Use the same configuration parameters as you used when configuring the service initially, namely federation service name (e.g. federation.domain.com), local admin details for the federation server and the federation certificate (unless you’ve replaced the certificate used, in which case obviously you should use the new certificate details); you noted those down during initial configuration, right?
  5. Once configuration is complete, the Remote Access Management Console should open automatically. All of your publishing rules should still be in place, and your published services should be available immediately.

For reference, here’s a sample config file, from which you should be able to reconstruct an appropriate file for your service:

<?xml version="1.0" encoding="utf-8"?> <configuration>   <configSections>     <section name="microsoft.identityServer.proxyservice" type="Microsoft.IdentityServer.Management.Proxy.Configuration.ProxyConfiguration, Microsoft.IdentityServer.Management.Proxy, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" />   </configSections>    <microsoft.identityServer.proxyservice>     <congestionControl latencyThresholdInMSec="8000" minCongestionWindowSize="64"       enabled="true" connectionTimeoutInSec="60" />     <connectionPool connectionPoolSize="200" scavengeInterval="5" />     <diagnostics eventLogLevel="15" />     <host tlsClientPort="49443" httpPort="80" httpsPort="443" name="federation.domain.com" />     <proxy address="" />     <trust thumbprint="1234567890ABCDEF1234567890ABCDEF12345678"       proxyTrustRenewPeriod="21600" />   </microsoft.identityServer.proxyservice>   <!-- <system.serviceModel>     <diagnostics>       <messageLogging logEntireMessage="true"               logMessagesAtServiceLevel="true"               logMessagesAtTransportLevel="true">       </messageLogging>     </diagnostics>   </system.serviceModel> --> </configuration>


Changing the Certificate on ADFS 3.0 and Web Application Proxy (WAP)

As with all systems using certificates for security, there comes a time when the certificate is expiring and needs to be replaced. here’s the procedure for ADFS 3.0 and WAP:

Starting with the ADFS server:

  1. Log onto the ADFS server.
  2. Add the new certificate to the server. Make sure this is added to the personal certificate store for the computer account. I usually do this using the certificates snap-in in MMC.
  3. Find the thumbprint for the new certificate. This can be found by looking at the details for the certificate; the thumbprint is usually at/near the bottom of the list of details for the certificate and consists of 40 hexadecimal characters. Take a copy of the thumbprint and ensure that the spaces are removed, so it’s a 40 character string; you’ll need this in a few moments.
  4. Grant the service account that is running the ‘Active Directory Federation Services’ service read access to the private key. To do this, follow these steps:
    1. Within the certificates snap-in of MMC, right click the certificate, select ‘All Tasks’ and then select ‘Manage Private Keys…’:
      Manage private keys
    2. Click ‘Add…’ to add the user account running the ADFS service on the server and grant read access to that user. Click OK on the permissions dialog to close it.
  5. Launch AD FS Management, expand ‘Service’ within the left pane and click ‘Certificates’:
    AF FS Manager Certificates
  6. Click ‘Set Service Communications Certificate…’ from the actions panel at the right of the screen:
    Set Services Communication Cert
  7. A dialog is shown presenting the available certificates on the server. Select the new certificate that is to be used. If you are unsure of the correct certificate, select each certificate in turn and click the ‘Click here to view certificate properties’ link which is shown and compare the thumbprint with that recorded earlier. Click OK on the dialog once the correct certificate is selected.
  8. If at this point you restart the server or ADFS service and make a connection to ADFS, you will still be presented with the original certificate. The change in the GUI changes the configuration in the ADFS configuration database, but not the certificate bound to HTTP.sys.
  9. To complete the configuration change, the following PowerShell command must be run:
    Set-AdfsSslCertificate –Thumbprint 00112233445566778899aabbccddeeff00112233
    Where 00112233445566778899aabbccddeeff00112233 should be replaced with the thumbprint you found earlier.
  10. Restart the server, or the ADFS service on the server to complete the configuration change.

Additional configuration is required on the WAP server:

  1. Log onto the WAP server.
  2. Add the new certificate to the server. Make sure this is added to the personal certificate store for the computer account.
  3. Run the following PowerShell command to change the certificate:
    Set-WebApplicationProxySslCedrtificate –Thumbprint 0011223344556677889900aabbccddeeff00112233
    Where 00112233445566778899aabbccddeeff00112233 should be replaced with the thumbprint you found earlier.
  4. All of the publishing rules need to be updated with the thumbprint of the new certificate (you created these originally using PowerShell, right?). This can be done by either deleting the old rules and recreating them with the new certificate thumbprint specified, or the rules can be updated with the new thumbprint, for example:
    Get-WebApplicationProxyApplication –Name “WebAppPublishingRuleName” | Set-WebApplicationProxyApplication –ExternalCertificateThumbprint “00112233445566778899aabbccddeeff00112233”
    Where (you guessed it!) 00112233445566778899aabbccddeeff00112233 should be replaced with the thumbprint you found earlier and ‘WebAppPublishingRuleName’ should be replaced with the name of the rule as it is shown in the Remote Access Console.
    I expected the federation publishing rule that was created automatically when WAP was originally configured to be updated for me, but had to manually switch the certificate on that one.
  5. Restart the server, or the ADFS and Web Application Proxy services to complete the configuration.
  6. Test that all of the previously published rules function correctly and provide the new certificate to the computer from which you are making a connection. If you need to check the certificate assigned to a specific publishing rule, the following PowerShell will show all of the properties for the publishing rule:
    Get-WebApplicationProxyApplication –Name “WebAppPublishingRuleName” | fl
    Note that the other parameters shown in the list generated by the above can also be changed (with a few exceptions) using the Set-WebApplicationProxyApplication cmdlet.