When software attacks!

Thoughts and musings on anything that comes to mind

Fixing SharePoint 2007 IIS WAMREG DCOM 10016 activation errors on Server 2008 R2

Anybody who works will SharePoint will grumble if you mention DCOM activation permissions. No matter how hard we try, how many patches we install (or how hard we try to ignore it), granting activation and launch permissions to the SharePoint service accounts is like plugging a dike with water-soluble filler.

On Server 2008  R2 our job is made that much harder by the fact that, by default, even administrators can’t edit the security settings for the IIS WAMREG service (GUID {61738644-F196-11D0-9953-00C04FD919C1}, for when you see it in your application event log).

The fix is to change the default permissions on a registry key, which you can only do by taking ownership of the key. My only comment would be that those permissions were locked down for a good reason in Server 2008 R2 and it’s somewhat frustrating that we need to do this.

Anyway, the key you are looking for is:

HKEY_CLASSES_ROOT\AppID\{61738644-F196-11D0-9953-00C04FD919C1}

To change the ownership you need to click the Advanced button in the Permissions tab of the properties dialog, then select the Owner tab. I’d recommend changing the owner to the Administrators group rather than a specific user, and make sure the permissions for TrustedInstaller are the same after you finished as they were before you started.

Once done, you can edit the DCOM permissions for the IIS WAMREG service in the same way as on other versions of Server 2008.

Twitter clients: Twinbox and Tweetz

Anybody who follows me on twitter will know that @rikhepworth is by no means a prolific tweeter. However, I do follow a number of people around the planet, and in addition to the ubiquitous Tweetie2 on my iPhone, I have found two clients to be useful and reliable.

The first is Tweetz, from Blue Onion Software. This is a great gadget for the Windows 7 desktop (or Vista Sidebar). The UI is simple and extremely usable (I love the way I can scroll the history for older tweets) and it makes posting a breeze.

The second reflects just how much I live by Outlook and the resulting ability to search and collate unread mails, blog posts and now tweets. Twinbox from TechHit allows you to tweet directly from Outlook and incoming tweets are collated by sender. No integration with the Office 2010 fluent UI but the add-in works, and there is a 64-bit version available as well.

Solve ‘pending reboot’ setup show stopper for CRM 4 Client (with Update Rollup 7)

I’ve been extremely busy over the past week creating demo systems and updating our own internal Black Marble systems. Part of that long list of tasks was to get around to testing the CRM 4 Outlook client with Outlook 2010.

For those who don’t know, you need the Update Rollup7 client if you want to use Outlook 2010 (and x86 Office only need apply). You can download a slipstreamed client installer from Microsoft.

However, you may find that the client steadfastly refuse to install, telling you that it is unable to proceed due to a pending restart.

The solution to the problem can be found on the Microsoft forums:

Look in the registry, in the Current User hive (HKEY_Current_User) for the user you’re trying to run setupclient.exe as. You will find a key in HKCU\Software\Microsoft named MSCRMClient. Create a new Dword value (32-bit if you’re on Windows 7 x64) called IgnoreChecks and set the value to 1.

This fixed it for me. Hopefully it will fix it for you too.

SharePoint Search Gatherer Error 10032

We encountered a problem recently with a two server farm. One server was configured as index and query server. Both servers were delivering pages to users. If a user executed a search on the server which did not run the search services, the Search page returned an error, and we saw the following in the application log and SharePoint logs:

Event Type:    Error
Event Source:    Office Server Search
Event Category:    Gatherer
Event ID:    10032
Description:
Could not create a database session.
Context: Application '2bee214b-e0b9-413b-8d85-c71002287e99'
Details:
    The database connection string is not available.   (0xc0041228)

Application 2bee214b-e0b9-413b-8d85-c71002287e99: The parent farm application root doesn't exist or access denied. - File:d:\office\source\search\search\searchdll\resourcemanagerimpl.cpp Line:703
Application 2bee214b-e0b9-413b-8d85-c71002287e99: Database session creation error for resource type 0. - File:d:\office\source\search\search\searchdll\resourcemanagerimpl.cpp Line:555

After a great deal of investigation and assistance from the guys at Microsoft, we identified the fault. On the server which had no search services configured, we added the following registry keys, copied across from the server which was running index and query functions:

HKLM\SOFTWARE\Microsoft\Office Server\12.0\Search\Applications\<SEARCH APPLICATION GUID>\ResourceManager                              Data=Server=sqlserver;Database=MOSSWEB_MYSITE_SSP_SEARCH;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15                              Server=sqlserver;Database=MOSSWEB_MYSITE_SSP_SERVICE;Trusted_Connection=yes;App=Windows SharePoint Services;Timeout=15

Where the Search Application GUID was the GUID from the error message, and the DB connection strings were copied over from the working server.

This resolved the problem, and now both servers deliver search results correctly.

I’m posting this here because I found only two possible leads in all my searching, neither of which detailed the fault. Hopefully, as always, this will help somebody else.

Kerberos for SharePoint on Server 2008 with IIS 7

UPDATE: Spence posted a great comment pointing out some issues with this post. Richard then restored our Community Server DB to a point in time before the post, so it’s been wiped. Post again, Spence, please, as I didn’t get chance to copy the text of the comment, I’m afraid.

I’ve not been doing so well with blog posts lately. I have more than one currently in process but unposted, and I just can’t seem to get the time to finish them – so apologies, CSW, for not getting the article I promised up yet, but I am working on it.

However, I needed to write up the work I did on our SharePoint at the end of last week, which I thought warranted being made available to a wider audience, so this a quick but hopefully helpful post.

Kerberos, Service Principal Names and Application Pool Identities

I’ve been migrating our SharePoint farm from Server 2003 to Server 2008, and because we now also use Microsoft CRM and a few other systems that require it, I’ve been configuring kerberos.

In theory, this should be simple: We always create service accounts in the AD for each web application to run as, so each of those accounts needs the correct SPN’s creating to match the web site.

For example, if our internal domain is mycorp.com and our SharePoint site is Portal running as the portalapp account, then I would register the SPNs of http/portal.mycorp.com and http/portal against the portalapp account using either adsiedit or setspn. I then make sure that the account is trusted for delegation, which I can do through the delegation tab in the account properties dialog in Active Directory Users and Computers. I also make sure that the servers running SharePoint are trusted for delegation to any service in the same way. UPDATE: Spence pointed out that this is completely unnecessary, see the comments, below.

Once I’ve done all that, I can enable Kerberos on the SharePoint web application through Central Administration. If you’ve never done that, the Authentication Providers option is in the Application Security section (usually the right hand column) in Application Management. Make sure you have the correct web application selected and choose the zone you want to configure (if you haven’t extended your web application, that’ll be default). In the Edit Authentication page, simply tick Integrated Windows Authentication and toggle the radio button beneath to Negotiate (Kerberos). Apply the changes, and we’re done.

Or so you’d think…

To be fair, with Server 2003, that should be it. With Server 2008, however, things just didn’t seem to be working properly for me. So I consulted the Oracle (on a side note, I’m trying a new Oracle lately…).

Kernel-mode authentication. Great idea, shame about the configuration

It turns out the IIS 7 has changed the way it deals with authentication, in that it now executes authentication-related processes in kernel mode for security and performance. That’s all well and good, but it also transpires that because of that, it uses the Local System account for this, and that’s where we hit a snag: I’ve created the SPN’s on the wrong account – I would need to create them on the machine account for the hosting server. Except that won’t work if we’re using more than one server in our farm to host the web applications, because I can only set the SPN against a single account.

It turns out that there is a solution to this. Frustratingly, however, it can’t be done through IIS Manager (or at least, I couldn’t see a way – perhaps Andy Westgarth and the IIS boys can help me here?). Once again we need to edit the applicationHost.config file, just like we did for the bindings, previously:

  1. Finding the right section for this can be tricky. You’re looking for the <location> section for your site, which then has a <system.webServer> section within it. I search on the site name (for example, our web site in IIS is SharePoint – Portal) because the line should look something like:
    <location path=”SharePoint – Portal”>
  2. Scroll down until you find the <security> section. In there you should see an <authentication> section and beneath that, <windowsAuthentication>. It will probably say:
    <windowsAuthentication enabled="true">
  3. Edit that line to read:
    <windowsAuthentication enabled="true" useKernelMode="true" useAppPoolCredentials="true">

You’ll need to do an IISReset after that, or at least I did.

Am I the only person that thinks this should be a setting in the GUI somewhere – it’s such a fundamental issue if you’re using any kind of farm-based system (such as SharePoint or CRM) that I can’t believe it’s so hidden.

UPDATE: Spence also pointed out that appcmd lets you configure this. I’ll post more when I’ve learned how to do it myself.

Appcmd syntax and a hotfix

After Spence posted his comments I did more digging. The syntax for appcmd to make the change I describe above is:
appcmd set config “SharePoint – Portal” /section:windowsAuthentication /useAppPoolCredentials:true /commmit:MACHINE/WEBROOT/APPHOST
where you need to replace the stuff in quotes with the name of your site. You can get a list using appcmd:
appcmd list site

I also found a note about a hotfix related to this issue. If you see your server suffering from blue screens after configuring kerberos (I haven’t… yet) then this might help.

Finally, Spence posted a link to a set of useful slides covering just this topic – thanks Spence, I bet those were three great sessions. Hopefully I’ve now corrected the errors you pointed out and this post is back to being helpful!

Configuring IIS Bindings to include host headers with https on Windows Server 2008 (for SharePoint)

NOTE: We use a wildcard SSL certificate which makes our life much easier when dealing with multiple hostnames. I have not tested this approach with multiple SSL certificates for specific sites.

We’ve been reconfiguring our SharePoint 2007 farm over the past couple of days and it’s now hosted on Windows Server 2008 and using NLB (network load balancing). The load balancer has been configured with a single public IP address and all our previous DNS CNAME registrations have been replaced with hostname A registrations pointing at the address. With our previous configuration we had multiple IP addresses on the server, one for each web application. Each IIS web site was then configured with a host header and ip address to allow for secure traffic over HTTPS.

With our new configuration, I didn’t want to specify an IP address on the web site. Handily, IIS 7 makes that scenario possible (and even relatively straightforward). The only snag is that you can’t configure the necessary bindings through the IIS Manager GUI. You can do it through an xml config file, however:

  1. Look in c:\windows\system32\inetsrv\config and edit the applicationHost.config file. Make sure you take a backup first!
  2. Find the <sites> section in the file. In there you will find a <site> element for each IIS web site. Each of those has a <bindings> element with each port/protocol binding listed. Our main site looked like this:
    <bindings>
    <binding protocol=”https” bindingInformation=”*:443:” />
    </bindings>

    and we changed it to look like this:
    <bindings>
    <binding protocol=”https” bindingInformation=”*:443:myhost.mydomain.com” />
    </bindings>
  3. Repeat for each web application. If you have more than one web application on the same IP address using either http or https you need to configure a host header or you’ll have problems.
  4. Execute an iisreset.

We now have all our content web applications, the SSP and the central administration web sites all running on a single IP address, many on the same port and using SSL.

As I said at the start of this post, we use a wildcard certificate which makes my initial IIS configuration easier. I haven’t tried multiple certificates, and I’m interested to know if that works or not.

Incoming Email with SharePoint on Windows Server 2008

I’ve been meaning to write this up for a while, simply because it’s not quite as straightforward as with Server 2005.

To configure incoming email on SharePoint when running on Server 2008 you’ll need to run through the following steps:

  1. Install the SMTP feature
    Open Server Manager. Click on Features in the left hand column then click add features in the right hand pane. Tick the SMTP Server check box and click install.
  2. Configure the SMTP Service in IIS Manager (version 7)
    Start Internet Information Services (IIS) Manager from Administration tools in the Start Menu. Once open, click the name of the web server to bring up the options in the centre panel. In the centre panel, right-click SMTP E-mail and select Open Feature from the menu.
    Click the option to ‘store e-mail in pickup directory’ and set the path to be c:\inetpub\mailroot\Drop (that’s the default).
  3. Configure the SMTP Service in ISS Manager (version 7)
    Start Internet Information Services (IIS) 6.0 Manager from Administration tools in the Start Menu. Expand the server to show the SMTP service. In the ‘domains’ section, add any email domain aliases you need in there. Configure the other SMTP service settings just like you did with Server 2005.

SharePoint Service Pack 2 Pains

I finally bit the bullet and decided to upgrade our SharePoint farm yesterday. I’d been holding off for a while because of time constraints and because of a known issue with Project Server, also part of our farm.

I took careful steps to increment the farm from the SP1+Infrastructure update all the way through each CU up until the service pack. That all worked fine. It was when I tried SP2 I hit problems.

The first issue was that once I’d installed the WSS patch, the Sp2 patch refused to install. Rebooting the server then caused chaos as all my services complained that the SharePoint DB was the wrong version (too old, because I hadn’t run the config wizard yet).

Andy and I spent a long time poking the server yesterday, and spent time building virtual machines to take over the farm as well. We finally knocked it on the head just shy of midnight and left the server in the state it was, trying to start the upgrade installer.

When I got in this morning, the upgrade had installed. I’m guessing that the problems we were seeing were related to services starting and needing time to fail, and we simply hadn’t given them enough time to fail (mind you, the paranoid disk integrity check took a while…)

Much happier, I started the upgrade wizard. Which promptly failed. The logs showed the following:

[WebApplicationSequence] [ERROR] [5/27/2009 7:48:09 AM]: Action 12.0.4.0 of Microsoft.SharePoint.Portal.Upgrade.WebApplicationSequence failed.
[WebApplicationSequence] [ERROR] [5/27/2009 7:48:09 AM]: Feature '20477d83-8bdb-414e-964b-080637f7d99b' is not installed in this farm, and can not be added to this scope.
[WebApplicationSequence] [ERROR] [5/27/2009 7:48:09 AM]:    at Microsoft.SharePoint.SPFeatureCollection.AddInternal(Guid featureId, SPFeaturePropertyCollection properties, Boolean force, Boolean fMarkOnly)
   at Microsoft.SharePoint.SPFeatureCollection.Add(Guid featureId, Boolean force)
   at Microsoft.SharePoint.Portal.Upgrade.ActivatePublisingTimerJobsWebAppFeature.Upgrade()
   at Microsoft.SharePoint.Upgrade.SPActionSequence.Upgrade()
[ActivatePublisingTimerJobsWebAppFeature] [12.0.4.0] [DEBUG] [5/27/2009 7:48:09 AM]: Begin Rollback()
[ActivatePublisingTimerJobsWebAppFeature] [12.0.4.0] [DEBUG] [5/27/2009 7:48:09 AM]: End Rollback()
[ActivatePublisingTimerJobsWebAppFeature] [12.0.4.0] [DEBUG] [5/27/2009 7:48:09 AM]: Begin Dispose()
[ActivatePublisingTimerJobsWebAppFeature] [12.0.4.0] [DEBUG] [5/27/2009 7:48:09 AM]: End Dispose()
[ActivatePublisingTimerJobsWebAppFeature] [12.0.4.0] [DEBUG] [5/27/2009 7:48:09 AM]: Elapsed time: 00:00:00.0312496.

A quick dig with our old friend google turned up a couple of similar posts from Jukka on Moss and MySharePointofView so I had a look at the 12 hive and to my surprise found that there was no folder for the PublishingTimerJobs feature. I copied it from one of the new servers, already patched to SP2 and ran the command:

stsadm –o installfeature –name PublishingTimerJobs

That succeeded. I then followed with the old favourite:

psconfig –cmd upgrade –inplace b2b –force

That upgrade has just completed. No project-related errors, just success, so I add my experience to the collective.

Creating a new Virtual PC using the Virtual Windows XP Base Disk

One of the most useful elements of the Virtual Windows XP feature in Windows 7 is that the VPC is easily replicated and you can have multiple virtual machines all publishing applications which run in their own sandboxes.

  1. Create a new Virtual Machine
  2. Create a Differencing Hard Disk from the Virtual Windows XP Base
  3. Start the VM and run through the setup wizard:
    1. Accept the Licence Agreement
      image
    2. Set the keyboard and locale to your needs
      image 
    3. Give the PC a name and administrator password
      image
    4. Set the time zone
      image
    5. Wait while it configures networking…
      image
    6. … and runs through the final steps, followed by a reboot.
      image
  4. Configure the VPC for updates and user accounts:
    1. On restart, choose an option for automatic updates
      image
    2. You should now be logged in as administrator
      image 
    3. Open up Computer Management and enable the ‘User’ account, then reset the account password to something you know.
      image
      image
    4. Enable Integration Features from the VPC Tools Menu
      image
    5. Set the login account to the user account you just enabled.
    6. Accept the logon message to disconnect Administrator
      image
  5. Configure the applications on the VPC:
    1. Once you’re logged on as User, create a new shortcut in c:\documents and settings\all users\start menu and wait a few minutes.
      image
      You should see your start menu update with the new application shortcut
      image
      Each virtual machine gets a folder in your start menu beneath Windows Virtual PC and the applications on each PC appear in there.
    2. Once you’ve finished configuring your applications, log off your session on the virtual PC (don’t close the PC or shut it down)
      image
    3. Then close the VPC down from the Action menu and choose Hibernate
      image

If you now start any of the applications that have appeared in your main computer’s Start menu, the VPC will fire up in the background and you application will appear on your desktop. This is a great way to create multiple VPCs with applications that might conflict with each other.

There is a catch, however. Windows Virtual PC requires hardware virtualisation support to work. In my opinion this is a mistake. Since the virtual machines use emulated hardware rather than accessing the machine hardware like Hyper-V VMs do, I can’t see the reasoning here. Virtual PC 2007 used the hardware virtualisation if it was available but didn’t force it on you, which was the correct approach. Lots of businesses will find this technology useful, but will discover that the majority of their computers won’t be able to use it. At that point, the solution may as well not exist, and I for one hope that Microsoft change their mind about hardware virtualisation support before Windows Virtual PC ships.