SharePoint Crawl Rules Appears to Ignore Some URL Protocols

I recently came across an issue relating to crawling people information in SharePoint and the use of crawl rules to exclude certain content.

The issue revolved around a requirement to exclude content contained within peoples’ MySites, but include user profile information so that people searches could still be conducted. The following crawl rule had been configured and was successfully excluding MySite content, but was also excluding the user profile data (crawled using the sps3s:// protocol):

URL Exclude or Include
https://mysite.domain.com/* Exclude

Using the crawl rule test facility indicated that while SharePoint treats http:// and https:// differently, https:// and sps3s:// appear to be treated the same as far as crawling is concerned, so if the above crawl rule is in place, items in the MySite root site collection, both with an https:// and sps3s:// prefix, will not be crawled, and therefore user profile data and people search will not be available:

Crawl rule test

[Screen shot from lab SharePoint 2010 system. however the same tests have been performed against SharePoint 2013 and 2016 with the same results]

In fact what is happening is that the sps3s:// prefix tells SharePoint which connector to use, and in the case of people search, this is translated into a call to a web service at the host specified, i.e. https://mysite.domain.com/_vti_bin/spscrawl.asmx, so the final call that is made is in fact to an https:// prefix, hence the reason that the people data is not crawled.

Replacing the above crawl rule with the following rule corrects the issue allowing people data stored in the MySite root site collection to be indexed and therefore be available for users to search:

URL Exclude or Include
https://mysite.domain.com/personal/* Exclude

Hiding a Default SharePoint 2010 Content Type Field

Extending some of the default content types available in SharePoint 2010 can result in unwanted fields being shown. An example is extending the ‘Event’ default content type, which has three fields attached which you don’t seem to be able to edit:

Unwanted event fields

In particular, many users would like to remove the Workspace field from calendar entries:

Workspace field

Some users resort to editing the new/edit/view forms to remove the fields that are not wanted, however there is an easier way!

Using the following PowerShell, an unwanted field can be hidden so that it does not appear in the new/edit/view forms:

1 # Get a reference to the web we are using 2 $web = Get-SPWeb https://intranet.domain.com/site 3 4 # Get a reference to the list to which the content type is attached 5 $list = $web.Lists["Holiday Calendar"] 6 7 # Return a list of the fields 8 $fields = $list.fields 9 10 # Select the field we wish to hide 11 $field = $fields | where {$_.internalname -eq "WorkspaceLink"} 12 13 # Show the current 'hidden' status of the field 14 $field.Hidden 15 16 # Set the field to hidden (note that 'CanToggleHidden' must be true to allow this) 17 $field.Hidden = $true 18 19 # Update the field 20 $field.Update()

Creating a new holiday event on the calendar, the field allowing a user to create a workspace is now hidden:

New item form without workspace field

SharePoint 2010 Service Application Communication Scheme

The default communication scheme for many of the SharePoint 2010 Service Applications is ‘http’ (i.e. unsecured). This can be changed easily in the GUI by selecting the service application and clicking the ‘publish’ ribbon button:

Service Application publish ribbon toolbar button

It should however be noted that a number of Service Application communication schemes run by default over https and cannot be modified, these are:

  • Application Discovery and Load Balancer Service Application
  • Search Administration Web Service
  • Secure Store Service Application
  • Security Token Service Application

The communication scheme of a few Service Applications cannot be inspected using the GUI (and the publish ribbon button remains greyed out when they are selected):

  • SharePoint Server ASP.NET Session State Service
  • SharePoint Session State Service Application
  • WSS_UsageApplication

Modifying the communication scheme of all of the Service Applications can be time consuming and can be error prone, especially when using SharePoint 2010 Enterprise and the Office Web Applications with all of the Service Applications available configured for use. With this in mind, the following PowerShell will change the communication scheme of all of the Service Applications where it is possible to do so to https:

1 # This script sets the communication scheme of all Service Applications to be https instead of http 2 # Note that the communication scheme for a number of Service Applications cannot be changed 3 4 # Grab a list of the Farm's Service Applications 5 $ServiceApps = Get-SpServiceApplication | Sort-Object TypeName 6 7 # Iterate through the Service Applications 8 foreach ($ServiceApp in $ServiceApps) 9 { 10 if (($ServiceApp.TypeName -ne "Application Discovery and Load Balancer Service Application") ` 11 -and ($ServiceApp.TypeName -ne "Search Administration Web Service Application") ` 12 -and ($ServiceApp.TypeName -ne "Security Token Service Application") ` 13 -and ($ServiceApp.TypeName -ne "Secure Store Service Application") ` 14 -and ($ServiceApp.TypeName -ne "SharePoint Server ASP.NET Session State Service") ` 15 -and ($ServiceApp.TypeName -ne "State Service") ` 16 -and ($ServiceApp.TypeName -ne "Usage and Health Data Collection Service Application")) 17 { 18 # We can modify the communication scheme 19 Write-Host "Current communication shceme for" $ServiceApp.DisplayName ":" $ServiceApp.DefaultEndpoint.Name 20 if (($ServiceApp.DefaultEndpoint.Name -eq "https") -or ($ServiceApp.DefaultEndpoint.Name -eq "secure")) 21 { 22 Write-Host "Service Application already using https, skipping" -ForegroundColor Red 23 } else { 24 # Change the communication scheme to https 25 26 27 if ($ServiceApp.TypeName -eq "PowerPoint Service Application") { 28 # PowerPoint Service Application has "fast" instead of "http" and "secure" instead of "https" 29 $SAEhttps = $ServiceApp | Get-SPServiceApplicationEndpoint | where {$_.DisplayName -eq "secure"} 30 } else { 31 $SAEhttps = $ServiceApp | Get-SPServiceApplicationEndpoint | where {$_.DisplayName -eq "https"} 32 } 33 Write-Host "Setting Service Application communication scheme to https" -ForegroundColor Green 34 $ServiceApp.DefaultEndpoint = $SAEhttps 35 $ServiceApp.Update() 36 } 37 Write-Host "`n" 38 } 39 }

To reverse these changes, and set the communication scheme of all Service Applications for which it is possible to modify the communication scheme, the following PowerShell can be used:

1 # This script sets the communication scheme of all Service Applications to be http instead of https 2 # Note that the communication scheme for a number of Service Applications cannot be changed 3 4 # Grab a list of the Farm's Service Applications 5 $ServiceApps = Get-SpServiceApplication | Sort-Object TypeName 6 7 # Iterate through the Service Applications 8 foreach ($ServiceApp in $ServiceApps) 9 { 10 if (($ServiceApp.TypeName -ne "Application Discovery and Load Balancer Service Application") ` 11 -and ($ServiceApp.TypeName -ne "Search Administration Web Service Application") ` 12 -and ($ServiceApp.TypeName -ne "Security Token Service Application") ` 13 -and ($ServiceApp.TypeName -ne "Secure Store Service Application") ` 14 -and ($ServiceApp.TypeName -ne "SharePoint Server ASP.NET Session State Service") ` 15 -and ($ServiceApp.TypeName -ne "State Service") ` 16 -and ($ServiceApp.TypeName -ne "Usage and Health Data Collection Service Application")) 17 { 18 # We can modify the communication scheme 19 Write-Host "Current communication shceme for" $ServiceApp.DisplayName ":" $ServiceApp.DefaultEndpoint.Name 20 if (($ServiceApp.DefaultEndpoint.Name -eq "http") -or ($ServiceApp.DefaultEndpoint.Name -eq "") -or ($ServiceApp.DefaultEndpoint.Name -eq "fast")) 21 { 22 Write-Host "Service Application already using http, skipping" -ForegroundColor Red 23 } else { 24 # Change the communication scheme to https 25 if ($ServiceApp.TypeName -eq "Visio Graphics Service Application") 26 { 27 # Visio Graphics Service Application has "" instead of "http" (equivalent to "default" in the GUI) 28 $SAEhttp = $ServiceApp | Get-SPServiceApplicationEndpoint | where {$_.DisplayName -eq ""} 29 } elseif ($ServiceApp.TypeName -eq "PowerPoint Service Application") { 30 # PowerPoint Service Application has "fast" instead of "http" and "secure" instead of "https" 31 $SAEhttp = $ServiceApp | Get-SPServiceApplicationEndpoint | where {$_.DisplayName -eq "fast"} 32 } else { 33 $SAEhttp = $ServiceApp | Get-SPServiceApplicationEndpoint | where {$_.DisplayName -eq "http"} 34 } 35 Write-Host "Setting Service Application communication scheme to http" -ForegroundColor Green 36 $ServiceApp.DefaultEndpoint = $SAEhttp 37 $ServiceApp.Update() 38 } 39 Write-Host "`n" 40 } 41 }

Note that an IISRESET will be required on all servers in the farm once either of the above PowerShell scripts has been run to complete the modification of the communication scheme.

Incorrect Title Shown for Office 2007/2010 Documents in SharePoint 2010 Search Results

Office 2007/2010 format documents stored in SharePoint 2010 which have their title field populated show an incorrect title in the default search results. The screen shot below shows the search results for the following document:

Filename: Test document 1.docx
Title: Test Document 1 Title
First line of document: First line of test document 1

Search results Title 01a

The title that is displayed as the link to the document at the top of the individual search result is the first line of text from the document, not the title (metadata) field. It should be noted that if the title field for the document is not set, the filename is displayed as the link to the document at the top of the individual search result, not the first line of text from the document.

Luckily, correcting this particular ‘feature’ is simple:

  • Open registry editor
  • Navigate to HKLM\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager
  • Edit the ‘EnableOptimisticTitleOverride’ key and modify its value to 0:
    Search results Title 02a
  • Restart the SharePoint Server Search 14 service by starting an admin command prompt and issuing the following commands:
    net stop osearch14
    net start osearch14
    Search results Title 04a
  • Repeat the above steps for all SharePoint servers in the farm.
  • Perform a full crawl on the SharePoint content source(s)

Once the full crawl has completed, performing the search again gives the title field as the search result title for a document which has the title field populated (note that in the screen shot below, test document 1 has the title field set, test document 2 doesn’t):

Search results Title 07a

The following can be copied into a .reg file to automate setting the key, or the key could be set using PowerShell:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager]
"EnableOptimisticTitleOverride"=dword:00000000

SharePoint 2007 with multiple domains

SharePoint can use multiple domains for user authentication, however I recently came across an issue when setting up an extranet using this scenario.

The steps for setting up SharePoint 2007 to use multiple domains for user authentication are relatively simple:

  • Add the second domain as a user profile source in the SSP
  • Issue the following stsadm command to encrypt the password for the account that is used to access the second forest or domain on each web front end server in the farm:
    stsadm –o setapppassword –password <password>
  • Issue the following stsadm command to add multiple domains to the people picker search list:
    stsadm –o set property –pn peoplepicker-searchadforests –pv domain:<original resource domain>;domain:<secondary domain>,<domain>\<username>,<password> –url <web application URL>

At this point I could successfully add users from the second domain to the security groups in a site collection or site, however when I attempted to log in as one of these users, I received a “500 – Internal server error”. Logging in as a user from the original resource domain worked fine however.

Modifying the web.config file for the web application to set CallStack to true and CustomErrors to Off didn’t give me any further information, at least in Internet Explorer 8, as I still saw the same “500 – Internal server error”, however viewing the web application in Firefox gave me a somewhat cryptic error:

Multi-domain login issue with FF - error

This error translates as STATUS_AUTHENTICATION_FIREWALL_FAILED, however the firewall wasn’t an issue in this scenario.

The solution was to grant the machine accounts for each of the web servers in the SharePoint farm an extra right in AD. The steps were:

  • On a domain controller, start “Active Directory Users and Computers”
  • On the view menu, ensure that the ‘Advanced Features’ option is checked
  • Locate the computer account for first of the web servers in the SharePoint farm
  • For the computer account, right-click and select properties and then click the security tab
  • Add the <external domain>\Domain Users group to the security list and grant the ‘Allowed to authenticate’ right. Click OK to close the dialog.
  • Repeat these steps for the computer accounts of all the other web servers in the farm

This resolved the issue and allowed users from the second domain to log into SharePoint.

Note that to achieve multiple domain authentication for SharePoint 2010, the same stsadm commands are used, and I therefore believe that the same issue may occur, however the above solution should also work for SharePoint 2010.

Publishing Access Services Database to SharePoint 2010 gives ‘<URL> did not respond…’

While building a portion of our demo SharePoint 2010 farm, I encountered an error when publishing an Access 2010 database to a SharePoint 2010 Access Services site.

The error which was shown was ‘<URL> did not respond. Either the server does not exist, Microsoft Access Services are not enabled on the server, or the server is using an older version of Microsoft Access Services which is not compatible with Access 2010’:

Access Services Publish Error

After checking that the Access Services farm and web application features were enabled, and that the enterprise features were enabled on the site collection to which I was attempting to publish the Access database (all were fine), I looked in the server application logs on the WFE servers in the demo farm, and on one of the farm servers saw the error ‘There is no default Access Services Application Proxy’:

Event Viewer Error

Checking the Service Application Associations (Central Administration –> Application Management –> Configure service application associations) showed that the Access Database Service proxy was not associated with the default proxy group:

Proxy Associations

After adding the Access Database Service proxy to the default group, publishing the Access database to SharePoint proceeded without a hitch.

In our case, the proxy not being associated with the default proxy group was due to us using PowerShell to configure the Access Service Application. If you do the same, check whether the proxy has been associated with the default group (or whatever proxy group you want it associated with).

Error during SharePoint 2010 Upgrade “Minimal master page failed to provision: could not find master page gallery”

During a recent SharePoint upgrade using the database attach method, I saw the following recorded in the upgrade error log:

[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: SPSite Url=http://intranet.domain.com
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: Minimal master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: SPSite Url=http://intranet.domain.com
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: V4 master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: SPSite Url=http://intranet.domain.com

Further information was provided in the upgrade log that accompanies the error log:

[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Setting UIVersion metadata on existing default master page
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Provisioning minimal master page in site: Site1
Site Url: http://intranet.domain.com/Site1
[Powershell] [PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: Minimal master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Provisioning V4 master page in site: Site1
Site Url: http://intranet.domain.com/Site1
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: V4 master page failed to provision: could not find master page gallery.
[PerWebUpgradeAction (4.0.10.0)] [INFO] [13/04/2011 15:55:48]: Setting UIVersion metadata on existing default master page
[PerWebUpgradeAction (4.0.10.0)] [WARNING] [13/04/2011 15:55:48]: Could not find master page gallery. Metadata on default.master not set.

The farm in question was originally a SharePoint 2003 farm that had been in-place upgraded to SharePoint 2007. The site recorded in the upgrade error log had been created in SharePoint 2003, but had not had its template selected at the time of original creation, allowing the first site user to pick the template to use. Critically however, this had never been done.

Once the database upgrade had been completed, navigating to the site gave me the expected screen to allow me to select the site template to use. Picking a template and clicking OK however produced a ‘file not found’ error.

The moral of the story seems to be ensure that any sites created without selecting a site template should either be completed, or deleted before the upgrade process starts. Note that you can still delete a site that has been upgraded to 2010 in the above state using PowerShell. Detection of sites such as these in SharePoint 2007 is straightforward using stsadm –o enumallwebs – any site that doesn’t have a template shown against it is in this state.

Finding the Current Site Template the Easy Way in SharePoint 2010

In SharePoint 2007 there seemed to be no easy way to work out which site template was used to create a site (particularly for the end user). Solutions existed using stsadm, or by uploading a custom page to SharePoint, but unless you were an admin, these routes are likely to be closed off to you.

Things are a little easier in SharePoint 2010!

To find the site template for the site you are viewing, follow these steps:

  • View the source for the page in your web browser
  • Search the source for the ‘g_wsaSiteTemplateId’ variable; it’s usually near the top of the source

The value of the variable is the site template that was used to create the site, for example:

var g_wsaSiteTemplateId = ‘STS#0’;

means that this particular site was created using the Team Site template.

Renaming the PerformancePoint Service Application database in SharePoint 2010

When creating the PerformancePoint Service Application, there is no way to control the name of the database that is created, not even when using PowerShell to create the Service Application. The database that gets created is in the form

<Service Application Name>_GUID

which for some reason a good many DBAs are not too keen on!

The database can however be renamed by following these steps:

  • Stop the PerformancePoint service on all SharePoint servers in the farm that are running the service using the ‘services on servers’ area of Central Administration:
    Stopping the PerformancePoint Service
  • Rename the database and log file – there are two ways of completing this; I prefer the second option of the two outlined below as it completely renames all of the references to the database, but it is a more involved process:
    1. Open SQL Server Management Studio on the SQL Server for the farm.
      Select the PerformancePoint Service Application database and then click again to allow renaming:
      Rename PerformancePoint DB in GUI
      Rename the database to match the naming convention you wish to use for farm databases. Note that this only renames the database friendly name as shown in SQL Server Management Studio and not the file names or the logical database and log file names.
    2. Alternatively:
      Open SQL Server Management Studio on the SQL Server for the farm.
      If you wish to, you can change the recovery mode of the PerformancePoint database to ‘simple’; this saves having to backup and restore a log file as well as the database file.
      Backup the PerformancePoint database created during the Service Application creation process.
      Restore the PerformancePoint database from the backup completed to a new database name which matches the naming convention you wish to use for farm databases. Note that the default naming convention for the log files on restore appends ‘_1’ to the database name to form the log file name; you may wish to change this to ‘_log’ to match the other log files that the database server hosts. The backup and restore will change the filenames used for the databases and the display name shown in SQL Server Management Studio, but not the logical database names. To change the logical database names, first find the logical names of the database and log for the database you wish to change; you can find this information either by taking note of the original database name when it was created, or from the ‘files’ section of the database properties screen within SQL Server Management Studio:
      Database Logical Names
      Execute the following two SQL queries:

      ALTER DATABASE <new PerformancePoint database name> MODIFY FILE (NAME="<original logical database name>", NEWNAME="<new PerformancePoint database name>")

      ALTER DATABASE <new PerformancePoint database name> MODIFY FILE (NAME="<original logical log file name>", NEWNAME="<new PerformancePoint database name>_log")

      If you changed the database recovery mode to ‘simple’, change it back to ‘full’.

  • On one of the SharePoint servers in the farm, open an instance of the SharePoint 2010 Management Shell, ensuring that it is run as administrator and issue the following PowerShell Commands:

    $newdatabasename = "<new PerformancePoint database name>"
    Set-SPPerformancePointServiceApplication -Identity "<name of the PerformancePoint Service Application>" -SettingsDatabase $newdatabasename

  • Restart the PerformancePoint service on the servers in the farm it was running on originally.
  • Delete the original PerformancePoint database that was created during the Service Application creation from SQL Server Management Studio.