Dealing with AD sync issues in an Azure hybrid deployment

I’ve been building demo environments for Tech.Days Online for the past few days. I had been blogging as I built, but then I hit problems and time pressure meant I had to pause my series on building the hybrid network. I will pick up the remainder of those posts in the near future but in the meantime, I want to give you all the heads up on one of my problems.

When I came to install the directory synchronisation tool on my azure-hosted dirsync server I couldn’t get the configuration wizard to run. It kept telling me that the domain admin user I entered (the domain administrator!) was not a valid domain account. I tried removing the machine from the domain and re-adding it first to no avail, and then started looking at domain replication.

There were no errors in the event log, save for one solitary warning on each DC about time. DCDIAG spat out huge lists of errors on both servers. Repadmin refused to admit that the domain controllers knew about each other.

So, for ref, I was seeing:

  • RPC errors when trying to talk between domain controllers.
  • DNS servers were not synchronising.
  • DCDIAG on one server threw up range of errors:
    • EventID: 0x80000B46 related to kerberos.
    • EventID: 0x00001695 relating to failed registration of DNS addresses in the domain.
    • EventID: 0x00001695 relating to failed registration of DNS addresses in the ForestDNSZones subdomain.
    • EventID: 0x00001695relating to failed registration of DNS addresses in the DomainDNSZones subdomain.
    • EventID: 0x0000168E relating to failed registration of the DNS record _ldap._tcp.ForestDnsZones.<domain>
    • EventID: 0x0000271A relating to a failed DCOM registration.

The solution was to fix time on the domain. I ran through a number of articles on the subject, and at the back of mind was a recollection that time can drift when running in a VM.

In the end I followed the settings in this TechNet article. I set both domain controllers to sync time with an NTP source (time.windows.com) and ten left the whole thing to settle for an hour or so while we went for food. When I came back the domain had settled.

Sadly, my dirsync box was a casualty of the conundrum. Because I removed it from the domain while AD sync was not working I had a broken trust relationship. I removed it for the domain again, rejoined it and everything worked just fine.

As always, hopefully this will save somebody some time and grief when doing the same thing as I was.

Building an Azure IaaS and on-premise hybrid environment Part 2: DC and servers in the cloud

This is part 2 of a series of posts bout building a hybrid network connecting Windows Azure and on-premise. For more background on what the goals are, and for information on how to create the Azure Network and connect the VPN tunnel between on-premise and cloud see part 1.

Creating a DC on our Azure Network

I’m going to create a new VM on Azure using the VM gallery. One important point when doing this is that you should add a second drive to the VM for domain controllers. This is down to how read/write caching works on the primary drive (it’s enabled)  which means there is a risk that a write operation may make it to the cache but not to the drive in the event of a failure. This would cause problems with AD synchronisation and for that reason we add a seond drive and disable caching on it so we can use it to host the AD database.

Before we create the new machine it’s a good idea to create a storage account. If we leave Azure to do it the account gets the usual random name. I prefer order and convention in these things, so I’ll create one myself.

storage 1

When you create a storage account, Azure now creates a container within it named vhds and it uses that to hold the virtual hard disks for your VMs.

We can now create a virtual machine using the VM Gallery.

new vm 1

The Virtual Machine creation wizard will appear and show the numerous VM templates we can start from. I want a Server 2012 R2 DC so I’m going to choose Windows Server 2012 R2 Datacenter from the list.

new vm 2

The next screen allows us to set the VM name. This is also used for the Azure Endpoint and must be unique within Azure. We can also choose a size for the VM from the available Azure VMs. This is a lab so I’m happy with a small VM. In production you would size the VM according to your AD.

We also need to provide a username and password that Azure will configure when it deploys the VM. We’ll use that to connect to the machine in order to join it to the domain.

new vm 3

The next screen asks for a whole bunch of information about where the new VM will be placed and what networks it will be connected to. The wizard does a pretty good job of selecting the right defaults for most settings.

I created two subnets in my virtual network so I could have an internal and external subnets. The DC shouldn’t have connections from outside our network so it’s going on subnet-1.

new vm 4

The final screen allows us to configure the ports that will be available through the Azure endpoints. If we remove these then we will only be able to connect to the new VM via our internal network. That’s exactly what I want, so I will click the big X at the right hand side of each endpoint to remove it.

new vm 5

When we click the final button Azure will show us that our new VM is provisioning.

new vm 6

Once the VM is running you can click on it to view the dashboard. You will see from mine that the new VM has no public IP address and that it has been give an internal IP address of 172.16.1.4 – on the Azure network I created earlier. The first server that you connect to a virtual network subnet in Azure will always get .4 as it’s address; the second gets .5, etc. An important point to note here is that if a virtual machine is deallocated (when you shut it down from the Azure portal it will do this) the DHCP-given IP address is released and another server could get that address. It’s important to be careful about the order you start machines in for this reason.

vm dash

 

I haven’t added a second hard disk to the VM, so that’s our next step. At the bottom of the dashboard there is an Attach button that allows us to add an empty disk to the VM.

attach disk 1

In the screen that appears we can give our new disk a name and size and, importantly, set the type of caching we want on the disk. As I mentioned, everything I have read and heard tells me that caching on the disk holding the AD database should be turned off.

new disk 1

Now we’ve got the second disk attached, the next step is to make an RDP connection to our new server. We can do that from one of the machines on our on-premise network just by entering the ip address of the Azure-hosted server into the Remote Connection dialog.

Remember to use the credentials you set when you created the VM: e.g. azureucdc\builduser

rdp connection 1

The first thing we need to do is bring the additional disk online, create a volume and assign a drive letter. I’ve used S for sysvol.

dc add disk

Next, we need to join the server to our AD domain, which will need a reboot. After that we can add the Active Directory Domain Services role in order to promote the server to be a domain controller. It’s important when doing this to set the paths for the AD databases to the second drive (S in my case)

azure dcpromo

Once we’ve got our new DC and DNS up and running, we should configure our Azure network so it knows the IP address of our new DNS and hands it to other servers in our network.

To do that we register the DNS with Azure first.

azure dns 2

Next we modify the configuration of our Azure virtual network to add the new DNS. The DNS addresses are handed out in the order they are specified in the Azure network, so I’ve removed the on-premise DNS then added first the one hosted in Azure and then the on-premise one.

azure dns 3

We now have a functioning Azure network with services that will support any other machines we host there even if the VPN link goes down.

We’ll need some more VMs for our other services to support our connected Azure ADFS. We’ll deal with those in part 3.

Building an Azure IaaS and on-premise hybrid environment Part 1: The plan and Azure Network Connection

I’ve been meaning to build a test lab to kick the tyres of Windows Azure Networks for a while. Two things combined together to make me get it done, however: First was the need to build exactly that for a customer as part of proof-of-concepts; the second was an invitation to present at Tech.Days Online on the subject.

I’ve built and rebuilt said lab a few times now. I am about to build it again in order to have a demo environment for Tech.Days and I though it would be a good opportunity to blog the steps involved.

I had planned this as one blog post, but it’s going to end up as a series of posts just because of sheer length. In this post I’m going to describe the environment and walk through creating the Azure network and connecting it to my on-premise network.

Constraints

Let’s get the big problem out of the way first: In order to do this you need a static, world-routable IP address. You have to have something at your end that will act as one end of a site-to-site VPN tunnel and that will only work if you have an IP that doesn’t randomly change.

The second issue is the bit of equipment you need to get that VPN working. The easiest is a Windows Server 2012 or 2012 R2 machine, or equipment from Juniper or Cisco which means you should be able to use one of the configuration scripts provided by Microsoft. After that, you’re on your own. I’ve already blogged about getting the connection working with a SonicWall. In theory, anything that supports IKEv2 connections can probably be made to work, but you’ll need to have a good understanding of the technology.

The third issue revolves around whether you are using trial subscriptions of things like Windows Azure and Office 365. I have an MSDN subscription to Azure, so it’s not going to vanish quickly, although I can burn through my MSDN credit quite quickly with this lab. If you are using a trial you need to be aware that you might hit problems. In this post I’m not going to touch Office 365, although I am going to get single sign-on working with Azure AD. The reason for that is I don’t want to connect my Azure AD to an Office 365 trial subscription that will vanish in a few short weeks.

What are we building?

I wanted to build a meaningful test of what Azure Networks could do for me. I turns out that it’s also something that interests a good many of my customers and colleagues in the industry:

  1. Create a Windows Azure virtual network and connect that to my on-premise network.
  2. Place a domain controller into the Azure network.
  3. Build a server in the Azure network to run Dirsync and push my domain users into Azure AD (step one of single sign-on for Office 365 as well)
  4. Build an ADFS server in the Azure network and connect it to Azure AD (step two of single-sign on for Office 365)
  5. Build an ADFS Proxy (or a Server 2012 Web Proxy) for internet-access to the federated sign-on mechanism (also needed for Office 365)

What aren’t we doing?

We’re not going to look at Azure Backup in this post. We already use Azure for backup at Black Marble as part of our DPM configuration. There is also a simpler backup client for Windows Server. I’ll  blog on those separately.

Why would we want to do this?

I have had many conversations about the practicality of moving virtualised servers from an on-premise environment into Azure. For many organisations this can work out cheaper that running their own tin. A hybird approach allows those VMs to me moved across over time and is important to enable testing.

Description of my rig

I am lucky in that I have a small number of static IP addresses through the internet provision at Black Marble. For my previous builds of this lab I added a USB network adapter to my laptop to connect to the outside world. For this build I am using a server hosted at Black Marble. Why? I need both ends of the lab to be running for my Tech.Days talks and it’s very hard to get a static world-routable IP at the event.

Very importantly, however, none of this lab will be connected to my main networks. I will create an internal Hyper-V network for the on-premise end of the lab and a second Hyper-V network that will connect to the outside world. The Hyper-V host will not be connected to any of these networks.

On my laptop I have Windows 8.1 and built all the VMs as Generation 2 virtual machines. My server, however, runs Server 2012 so the VMs are Generation 1. Does this matter? Not a bit, but I like the simplicity of the generation 2 virtual machines more than generation 1.

On-Premise VMs

For my on-premise network I have two virtual machines:

PremUCDC

Role: Domain Controller, DNS
OS: Windows Server 2012 R2
CPU: 2 core
RAM: Dynamic, min 512MB, max 2048MB
Disk: 120Gb Dynamic
Network: I adapter on internal network

Once the OS has been installed I will add the Active Directory Domain Services role and promote the server to be a domain controller.

For this lab I am using an internet-registered domain and I will use that as the AD domain suffix. This is an important point: If you create an AD forest with a .local suffix then you will have to jump through some more hoops to get single sign-on with Azure AD working.

PremUCRRAS

Role: Remote Access/VPN
OS: Windows Server 2012
CPU: 2 core
RAM: Dynamic, min 512MB, max 2048MB.
Networking: I adapter on internal network; I adapter on internet-facing network

I’ve tried Server 2012 R2 for the RRAS server a couple of times and I have not yet managed to get the VPN connection working, so this lab will use Server 2012.

I won’t add any roles to this server. Once I’ve configured my Azure network I will use the configuration script from Microsoft to add the necessary roles and configure the VPN link.

My on-premise network will use the 192.168.0.0 non-routable address space. I will then use 172.16.0.0 for my Azure network address space.

Azure Network

We need to create a new network in Azure to hold our VMs and we need to tell Azure what our on-premise network looks like, and what DNS servers we already have.

We start by creating a new Local Network in Windows Azure:

new internal network

We will be asked what to call the new network and we are also asked to provide the IP address of the machine at the on-premise end of the VPN tunnel.

internal network 1

Next, we are asked to specify what address spaces we have on our on-premise network. This is important as it will be used for routing traffic between Azure and our on-premise networks.

As I said earlier, my on-premise network uses the 192.168 address space. I configure the Azure local network as 192.168.0.0/16. I can still subnet that network down on-premise (and I will).

internal network 2

Now we’ve defined what our on-premise network looks like we need to register our DNS servers. We need to do this so that Azure can hand them out via DHCP and machines on the Azure network will then be able to communicate with our on-premise network systems.

new dns

Our on-premise network has a single DNS – PremUCDC, 192.168.1.1

new dns 2

Now we can define our Azure network

new internal network

First we give it a name, and we need to associate it with an affinity group.

azure network 1

Next, we associate the new network with the DNS server we added earlier

azure network 2

We know we want to connect our on premise network so next we select the Configure site-to-site VPN option. Azure will helpfully add the local network we configured earlier.

azure network 3

The next step is to define our address range and subnets on Azure. I’ve chosen to create 172.16.0.0/24 so I can subnet that down. I want two subnets – 172.16.1.0/24 and 172.16.2.0/24. I’ll add the gateway subnet in a moment.

azure network 4

The gateway subnet is that one that has the internal IP address of the VPN endpoint in Azure. I want that to be 172.16.254.0/24.

azure network 5

The network will take a little while to create. Once it’s done we need to create a Gateway that will provide our VPN connection.

The dashboard for the new network will look something like the image below and we can click the button at the bottom to create our gateway. There are two options for the gateway – static or dynamic routing. Four the lab I’m using dynamic routing.

gateway 1

Creating a gateway can take up to fifteen minutes so this is a good time for a coffee break.

Once the gateway is provisioned your dashboard will update to show you the IP address of the Azure end of the VPN and metrics for data in and out. I’ve removed the last two octets of my gateway address in the screenshot.

gateway 2

To bring up the connection to on-premise we need to configure our RRAS server. Conveniently, we can click the Download VPN Device Script link to grab some powershell to do that for us.

The VPN scripts are also available for a range of Cisco and Juniper devices. Simply select the appropriate options in the menus and download the script you need. You will need to edit the scripts to set a number of parameters before running them.

gateway 3

Before we make those script changes we need to know the shared secret that is used in the VPN handshaking. To find that, click the Manage Key button at the bottom of the dashboard.

The shared key will be displayed. Copy it to clipboard and save it somewhere.

gateway 4

Next, we need to edit the powershell script to add the parameters.

As a side note, why the guys at Microsoft didn’t put a variable block at the top of the script to make this easier I don’t know. Search and replace it is, then…

vpnscript 1

There are a number of edits we need to make:

  1. Replace <SP_AzureGatewayIpAddress> with the large IP address that’s on the Azure network dashboard. This applies to lines 75, 80, 81 and 87.
  2. Replace <SP_AzureVnetNetworkCIDR> with the Azure network address range (172.16.0.0/16 in my lab)
  3. Replace <SP_PresharedKey> with the key we access through Manage Key.

I usually leave line 87 commented out and run the command manually after the configuration is complete.

Copy that script onto the VM that’s going to be the RRAS server. I’m assuming that you’ve already joined it to the domain and configured the network adapters for internal and internet connections.

In order to run the script we will need to modify the execution policy on the server to allow unsigned scripts.

The command for this is set-executionpolicy unrestricted

Now run the powershell script to configure the VPN connection. The script will add the necessary roles to the server and configure the vpn connection.

vpnscript 2

Once the script is run, assuming you don’t need to restart the server it’s time to bring up the connection.

We need to enable the Azure gateway first by clicking the Connect button on the dashboard. Once that’s done, execute the last line of the powershell script (connect-s2svpninterface …).

We should be able to see the state of the connection from both the RRAS server and the Azure dashboard. On the RRAS server, open the Routing and Remote Access console.

The VPN is listed in Network Interfaces and the connection status should be Connected.

vpn state 1

The Azure Network dashboard should also show that the connection has been made.

vpn state 2

 

 

Next steps…

Now we’ve got our VPN connected the next step is to create a new VM on Azure, join it to our domain and promote it to be a DC and DNS server.

Once that’s done there are more VMs to create in order to configure single sign-on with Azure AD, which also needs to be configured to use the internet-registered DNS domain.

With the VMs in place we can configure directory synchronisation and then configure ADFS for single-sign on.

Each of these will be covered in later articles in this series.