SharePoint Crawl Rules Appears to Ignore Some URL Protocols

I recently came across an issue relating to crawling people information in SharePoint and the use of crawl rules to exclude certain content.

The issue revolved around a requirement to exclude content contained within peoples’ MySites, but include user profile information so that people searches could still be conducted. The following crawl rule had been configured and was successfully excluding MySite content, but was also excluding the user profile data (crawled using the sps3s:// protocol):

URL Exclude or Include
https://mysite.domain.com/* Exclude

Using the crawl rule test facility indicated that while SharePoint treats http:// and https:// differently, https:// and sps3s:// appear to be treated the same as far as crawling is concerned, so if the above crawl rule is in place, items in the MySite root site collection, both with an https:// and sps3s:// prefix, will not be crawled, and therefore user profile data and people search will not be available:

Crawl rule test

[Screen shot from lab SharePoint 2010 system. however the same tests have been performed against SharePoint 2013 and 2016 with the same results]

In fact what is happening is that the sps3s:// prefix tells SharePoint which connector to use, and in the case of people search, this is translated into a call to a web service at the host specified, i.e. https://mysite.domain.com/_vti_bin/spscrawl.asmx, so the final call that is made is in fact to an https:// prefix, hence the reason that the people data is not crawled.

Replacing the above crawl rule with the following rule corrects the issue allowing people data stored in the MySite root site collection to be indexed and therefore be available for users to search:

URL Exclude or Include
https://mysite.domain.com/personal/* Exclude

SPWakeUp for SharePoint 2016

If you use SharePoint, you’ll know that some mechanism to wake up the hosted sites after the application pools are recycled overnight is very helpful (essential even) for the end user experience.

I’ve compiled a version of SPWakeUp for SharePoint 2016, which can be downloaded from https://onedrive.live.com/redir?resid=439F1389F21A368F%21496648.

If you want to compile this for yourself, this is the method I followed to get the above version:

  1. Grab a copy of the source code for SPWakeUp from https://spwakeup.codeplex.com/downloads/get/152410 and unpack it.
  2. Open the solution in Visual Studio (I used Visual Studio 2015) and allow the automatic upgrade.
  3. Replace the reference to the Microsoft.SharePoint.dll in the solution with one pointing to the SharePoint 2016 version. You’ll want to grab a copy from C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI on a SharePoint 2016 server.
  4. Modify the target framework for the application. I used 4.6.1 for the build above.
  5. Build either the debug or release version.

SharePoint 2007 with multiple domains

SharePoint can use multiple domains for user authentication, however I recently came across an issue when setting up an extranet using this scenario.

The steps for setting up SharePoint 2007 to use multiple domains for user authentication are relatively simple:

  • Add the second domain as a user profile source in the SSP
  • Issue the following stsadm command to encrypt the password for the account that is used to access the second forest or domain on each web front end server in the farm:
    stsadm –o setapppassword –password <password>
  • Issue the following stsadm command to add multiple domains to the people picker search list:
    stsadm –o set property –pn peoplepicker-searchadforests –pv domain:<original resource domain>;domain:<secondary domain>,<domain>\<username>,<password> –url <web application URL>

At this point I could successfully add users from the second domain to the security groups in a site collection or site, however when I attempted to log in as one of these users, I received a “500 – Internal server error”. Logging in as a user from the original resource domain worked fine however.

Modifying the web.config file for the web application to set CallStack to true and CustomErrors to Off didn’t give me any further information, at least in Internet Explorer 8, as I still saw the same “500 – Internal server error”, however viewing the web application in Firefox gave me a somewhat cryptic error:

Multi-domain login issue with FF - error

This error translates as STATUS_AUTHENTICATION_FIREWALL_FAILED, however the firewall wasn’t an issue in this scenario.

The solution was to grant the machine accounts for each of the web servers in the SharePoint farm an extra right in AD. The steps were:

  • On a domain controller, start “Active Directory Users and Computers”
  • On the view menu, ensure that the ‘Advanced Features’ option is checked
  • Locate the computer account for first of the web servers in the SharePoint farm
  • For the computer account, right-click and select properties and then click the security tab
  • Add the <external domain>\Domain Users group to the security list and grant the ‘Allowed to authenticate’ right. Click OK to close the dialog.
  • Repeat these steps for the computer accounts of all the other web servers in the farm

This resolved the issue and allowed users from the second domain to log into SharePoint.

Note that to achieve multiple domain authentication for SharePoint 2010, the same stsadm commands are used, and I therefore believe that the same issue may occur, however the above solution should also work for SharePoint 2010.

Error during SharePoint 2010 Upgrade “Minimal master page failed to provision: could not find master page gallery”

During a recent SharePoint upgrade using the database attach method, I saw the following recorded in the upgrade error log:

[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: SPSite Url=http://intranet.domain.com
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: Minimal master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: SPSite Url=http://intranet.domain.com
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: V4 master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: SPSite Url=http://intranet.domain.com

Further information was provided in the upgrade log that accompanies the error log:

[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Setting UIVersion metadata on existing default master page
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Provisioning minimal master page in site: Site1
Site Url: http://intranet.domain.com/Site1
[Powershell] [PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: Minimal master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Provisioning V4 master page in site: Site1
Site Url: http://intranet.domain.com/Site1
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: V4 master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Setting UIVersion metadata on existing default master page
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: Could not find master page gallery. Metadata on default.master not set.

The farm in question was originally a SharePoint 2003 farm that had been in-place upgraded to SharePoint 2007. The site recorded in the upgrade error log had been created in SharePoint 2003, but had not had its template selected at the time of original creation, allowing the first site user to pick the template to use. Critically however, this had never been done.

Once the database upgrade had been completed, navigating to the site gave me the expected screen to allow me to select the site template to use. Picking a template and clicking OK however produced a ‘file not found’ error.

The moral of the story seems to be ensure that any sites created without selecting a site template should either be completed, or deleted before the upgrade process starts. Note that you can still delete a site that has been upgraded to 2010 in the above state using PowerShell. Detection of sites such as these in SharePoint 2007 is straightforward using stsadm –o enumallwebs – any site that doesn’t have a template shown against it is in this state.

Renaming the PerformancePoint Service Application database in SharePoint 2010

When creating the PerformancePoint Service Application, there is no way to control the name of the database that is created, not even when using PowerShell to create the Service Application. The database that gets created is in the form

<Service Application Name>_GUID

which for some reason a good many DBAs are not too keen on!

The database can however be renamed by following these steps:

  • Stop the PerformancePoint service on all SharePoint servers in the farm that are running the service using the ‘services on servers’ area of Central Administration:
    Stopping the PerformancePoint Service
  • Rename the database and log file – there are two ways of completing this; I prefer the second option of the two outlined below as it completely renames all of the references to the database, but it is a more involved process:
    1. Open SQL Server Management Studio on the SQL Server for the farm.
      Select the PerformancePoint Service Application database and then click again to allow renaming:
      Rename PerformancePoint DB in GUI
      Rename the database to match the naming convention you wish to use for farm databases. Note that this only renames the database friendly name as shown in SQL Server Management Studio and not the file names or the logical database and log file names.
    2. Alternatively:
      Open SQL Server Management Studio on the SQL Server for the farm.
      If you wish to, you can change the recovery mode of the PerformancePoint database to ‘simple’; this saves having to backup and restore a log file as well as the database file.
      Backup the PerformancePoint database created during the Service Application creation process.
      Restore the PerformancePoint database from the backup completed to a new database name which matches the naming convention you wish to use for farm databases. Note that the default naming convention for the log files on restore appends ‘_1’ to the database name to form the log file name; you may wish to change this to ‘_log’ to match the other log files that the database server hosts. The backup and restore will change the filenames used for the databases and the display name shown in SQL Server Management Studio, but not the logical database names. To change the logical database names, first find the logical names of the database and log for the database you wish to change; you can find this information either by taking note of the original database name when it was created, or from the ‘files’ section of the database properties screen within SQL Server Management Studio:
      Database Logical Names
      Execute the following two SQL queries:

      ALTER DATABASE <new PerformancePoint database name> MODIFY FILE (NAME="<original logical database name>", NEWNAME="<new PerformancePoint database name>")

      ALTER DATABASE <new PerformancePoint database name> MODIFY FILE (NAME="<original logical log file name>", NEWNAME="<new PerformancePoint database name>_log")

      If you changed the database recovery mode to ‘simple’, change it back to ‘full’.

  • On one of the SharePoint servers in the farm, open an instance of the SharePoint 2010 Management Shell, ensuring that it is run as administrator and issue the following PowerShell Commands:

    $newdatabasename = "<new PerformancePoint database name>"
    Set-SPPerformancePointServiceApplication -Identity "<name of the PerformancePoint Service Application>" -SettingsDatabase $newdatabasename

  • Restart the PerformancePoint service on the servers in the farm it was running on originally.
  • Delete the original PerformancePoint database that was created during the Service Application creation from SQL Server Management Studio.

‘User Cannot Be Found’ Error During Initial SharePoint Configuration Wizard Run

I recently saw the above error on a SharePoint 2007 install of a test system. Reading a few blog posts, it seemed that most people were reporting the error as an issue when there was a problem with communication between the SharePoint server on which the wizard was being run and the domain controller it was connecting to as its logon server.

My machines didn’t seem to have any issues talking to each other however. Just to make sure, I changed them both over to a private virtual network, but that didn’t seem to help at all.

After a bit more rummaging, it turned out that the batch file I had used to create the users I needed for the SharePoint service accounts on the domain controller hadn’t created the ‘user@domain’ format of the user account correctly.

Recreating the users manually immediately solved the problem and the wizard ran successfully.

Cannot remove folder “SharePoint site name”

I recently ran into this issue while trying to move a series of sub-sites to site collections on a SharePoint farm. In order to achieve part of the process, I’d scripted the deletion of the sub-site tree using the stsadm –o deleteweb command once I’d exported it as there were quite a number of sub-sites and I wanted to try and save a little time.

I’d see the error message ‘cannot remove folder “sub-site name”’ for a couple of the sub-sites at a particular level, then the next couple might well be removed correctly, then more ‘cannot remove…’ errors.

To diagnose the issue a little more, I tried using the alternate options for the deleteweb stsadm command, namely

stsadm –o deleteweb –webid <sub-site ID> –databasename <content web application DB name> –databaseserver <DB server> -force

Note: To find a list of the sub-site IDs, run the following command:
stsadm –o enumallwebs –databasename <content web application DB name>
This will produce a list of all of the sub-sites including their URL and ID

Using the alternate options for the deleteweb stsadm command provided me with a lot more information on what the issue was that was stopping me from deleting some of the sub-sites.  In may case, the error shown was ‘The transaction log for database <DB name> is full. To find out why space in the log cannot be reused, see the log_reuse_wait_desc column in sys.databases’.

Checking the autogrowth options for the log file for the content database in question did indeed show that the file size was limited to quite a small value. Increasing the autogrowth limit on the transaction log to a more reasonable size immediately got me up and running again.

Trying to download more than 50Mb from SharePoint 2007 via webdav? You’ll need to know this…

If you store large files in SharePoint 2007 and are using webdav to access them (Explorer View uses webdav for example), you may be seeing the following error:

Cannot Copy <filename>: Cannot read from the source file or disk

If so, this relates to a registry key which is set on the local machine to limit the maximum file download size to 50Mb when using webdav.  To correct this behaviour, change the value of the FileSizeLimitInBytes registry key needs changing on the client machines.  To do this, follow these steps:

  • Start regedit
  • locate the following registry key:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebCLient\Paramters
  • In the right pane of the registry editor, right click the FileSizeLimitInBytes key and select ‘modify’
  • Enter a new value for this maximum file size (in bytes, as the key states) and click OK.
  • Close regedit and restart the computer.

Instead of making this change individually on each of the client machines, this key can of course be distributed via your organisation’s group policy.

Microsoft SharePoint Conference 2009: Day 4

Today’s highlights from the sessions I attended:

  • One of the new features of SharePoint 2010 is Access Services – we at Black Marble have been doing some work using Access Services (see Richard’s blog post for more details, along with the Access 2010 demo on Channel9), which Robert and I presented during the first session of the day.  The talk was received well, with a number of people wandering over to me during the rest of the day to let me know they liked it.
  • The number of management packs for System Center Operations Manager required to monitor SharePoint 2010 has been reduced to precisely one!