This week will be a short tip about how to enable Kerberos logging. As per KB262177 (How to enable Kerberos event logging), Kerberos logging can be enabled by creating the following registry key:

• HKLM\SYSTEM\CurrentControlSet\Control\Lsa\Kerberos\Parameters
• Registry Value: LogLevel
  Value Type: REG_DWORD
  Value Data: 0x1

To disable it, simply delete the value or set it to 0. After enabling the logging you will receive Kerberos error event entries in the System event log. The setting will become effective immediately on Windows Server 2008, on Windows Vista, on Windows Server 2003, and on Windows XP. For Windows 2000, you must restart the computer.

When searching for Kerberos debug options, the registry keys “KerbDebugLevel” and “LogToFile” (also below the Lsa\Kerberos\Parameters hive) might come up. It is to my understanding that these do not have any effect on Windows 2008 and up. I asked this a while back on ActiveDir.org: Kerberos debug logging on Windows 2008

What events will be logged? The following is a short summary of the most common entries:

• KDC_ERR_PRINCIPAL_UNKNOWN: SPN not registered in AD, or service is being accessed under the wrong name
• KDC_ERR_PREAUTH_FAILED: can be safely ignored, is due to the way Kerberos works and how some Kerberos client compose their first request
• KRB_AP_ER_MODIFIED: bad error! someone is presenting a ticket for a service to a service while the ticket was encrypted with an account not responsible for the service. Wow what a line, bottom line: authentication is failing, no fallback to NTLM either!
• …

This MS document is a reference for the above (and other errors), contains many useful troubleshooting hints: Troubleshooting Kerberos Errors

This week I encountered a problem when trying to run an export for the FIM Management Agent (FIM MA). The error given in the Synchronization Manager was failed-creation-via-web-services and the stack trace given:

There is an error executing a web service object modification request.
Type: System.NullReferenceException

Message: Object reference not set to an instance of an object.

Stack Trace:    at MIIS.ManagementAgent.RavenMA.DoAttributeLevelExport(DataSourceObject dsObject, String objClass, UninitializedResource resource)
   at MIIS.ManagementAgent.RavenMA.ExportObjectModification(DataSourceObject dsObject, SchemaManager schemaManager)
   at MIIS.ManagementAgent.RavenMA.Export(DataSourceObject dsObject)

Inner Exception:

I had been exporting before just fine. What did change? I added a new attribute to the FIM schema and was now trying to flow some data into it. To extend the FIM schema with a new attribute I followed the following steps:

1. Add attribute to the schema (with attribute validation)
2. Add binding for that that attribute to the user object
3. Changed the following MPR’s: added the attribute to the target set of the MPR’s
   1. Administration: Administrators can read and update users
   2. Synchronization: Synchronization account controls users it synchronizes
4. Altered the filter permissions for administrators so they can use the new attribute

All I wanted to do next is flow some data from the MV to the FIM MA using the defined attribute flows on the FIM MA.

In the past the above error was shown in multiple cases:

1. Jorge de Almeida Pinto (Export to ILM “2” throws a ‘Failed-Creation-Via-Web-Services’ error): Attribute validation error: make sure the values being flowed are conform the validation regular expression (like ^(Admin|Service|Test|User)?$ will only accept Admin, Service, Test & User)
2. Jorge again: Security validation: make sure the Synchronization MPR is correctly configured
3. David Lundell (Object reference not set to an instance of an object): Using the wrong account in the FIM MA configuration or to refresh the FIM MA Schema: the FIM MA account is not the FIM Service account!

However none of the above were applicable for me. Brad Turner (FIMMA Export Error: Object reference not set to an instance of an object) solved it by restarting the FIM Synchronization Service. Didn’t work for me. As a real last resort I completely shut down my FIM environment (it’s a lab environment…): this includes both Portal/Services Nodes, the Active and Standby Sync server and the SQL server. After booting al components the export went fine… I really hope I don’t see this one again.

Back in April KB980360 was released. It had a nice description of the changes in the BPA for the Active Directory Domain Services. Some days ago the update was finally made available for download… And now the KB is gone. Weird.

The update can be downloaded here: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=45a9c434-163f-47ed-a285-60e1c0b988b9 

There is also a link to the knowledge base article but it’s not working as well.

I remember the BPA had some new checks regarding Service Principal Names (duplicate check amongst others)

For those interested: a webcast is coming up:

Attend this webcast to learn about real-world scenarios and best practices for deploying a Microsoft identity and access management solution, using Microsoft Forefront Identity Manager 2010, Microsoft Forefront Unified Access Gateway 2010, and Active Directory Federation Services. Discover how these technologies help you provide your users with more secure access to applications from virtually anywhere.

Presenter: Brjann Brekkan, Senior Product Manager, Microsoft Corporation

TechNet Webcast: Deploying a Microsoft Identity and Access Management Solution (Level 300)

When I was installing the FIM Portal & Service binaries on the second node of my NLB setup the setup failed. I went looking in the log file and it told me the disk was full. After starting up TreeSize (in UAC mode, or else the statistics are completely wrong ;) ) I quickly found the Sharepoint logs being the culprit.

The log files can be found at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\LOGS. Mine were about 30 GB and it took them about 1 hour to get that big… On a fresh WSS install, no activity whatsoever on the site…

The log file looks like this:

And the error:

06/14/2010 15:59:49.06 OWSTIMER.EXE (0x04F8) 0x0508 Windows SharePoint Services Timer 5uuf Monitorable The previous instance of the timer job 'Config Refresh', id '{EA314111-18D2-4E59-B709-56608E8D5BD4}' for service '{392FB345-12E6-4F93-8528-29136C9B2602}' is still running, so the current instance will be skipped. Consider increasing the interval between jobs.

There is no real solution, just a workaround… It’s explained in KB941789: You receive error messages after you install security update 931212 (MS07-040) on a Windows SharePoint Services 3.0 Web front-end server or on a SharePoint Server 2007 Web front-end server

Whereas the previous tip was mostly targeted at SQL related stuff, this one will go back to the basics. Whenever you’re doubting Service Prinicpal Name (SPN) registration, you can start using setspn. With each new version of Windows the setspn command line utility has been extended. The options below are based on the Windows 2008 R2 setspn.

• setspn –x: allows you to do a quick check for duplicate SPN’s in the domain. Which in turn might explain why you are falling back to NTLM

• setspn –l: allows you to list the registered SPN’s for a given machine or user account

• setspn –q: allows you to query for a given SPN

• setspn –d: allows you to remove a given SPN from a given account

• setspn –a: allows you to register a SPN for a given account: try to avoid this one, use setspn –s (and –f) instead.
• setspn –s: allows you to regsiter a SPN for a given account after verifying no duplicates exist in the domain
• setspn –f –s: allows you to regsiter a SPN for a given account after verifying no duplicates exist in the forest

Whenever registering SPN’s you have to carefully construct it: what service is it for, which name will be used to access it, and what port is it running at. For most services this is straightforward, but Internet Explorer as a web browsing client complicates this. IE6, IE7 and IE8 still ignore the port entered in the address bar. They even ignore the name if it is a CNAME record. Why is there a difference between a CNAME and A record?

In the screenshot you can clearly see that in the first case (A record) the “alias.home.local” resolves to an IP address. However with a CNAME record, the “cname.home.local” resolves to the “dc01.home.local” and then to the IP address. If you are browsing with IE to cname.home.local, a SPN will be queried for using “dc01.home.local”.

Therefore it’s advised to always use A records for your web sites. And keep in mind that registering the port in the SPN for HTTP web sites is mostly in vain. You can alter this behavior according to KB908209: Internet Explorer 6 cannot use the Kerberos authentication protocol to connect to a Web site that uses a non-standard port in Windows XP and in Windows Server 2003 It says IE6 in the title but in the applies to section IE7 and IE8 are mentioned as well. In case you really aren’t getting where you want with the SPN’s if you can’t include the port number you could create the registry keys as described in the KB article.

So as a final tip for today, make sure to use ping whenever troubleshooting your SPN’s. It will show you how and if the name you registered the service under is reachable.

A little bird (Ned Pyle) at AskDS told me that ADMT 3.2 is released!

The software can be downloaded here: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=20c0db45-db16-4d10-99f2-539b7277ccdb

One of the limitations of ADMT 3.1 was that it couldn’t be installed on Windows 2008 R2, ADMT 3.2 now can. Another limitation was that you had to install it on a domain controller to be able to migrate sidHistory using the command line. I hope they solved this issue. This and other issues are described more in detail at KB976659: Known issues that may occur when you use ADMT 3.1 to migrate to a domain that contains Windows Server 2008 R2 domain controllers

When you’re encountering weird errors you might try enhancing the logging to find out which component is bothering you. This post will provide two ways to get some additional information.

An example I have encountered the Service Not Available error.

The event viewer is helpful, but it does not go into detail:

Log Name:      Application
Source:        Microsoft.ResourceManagement.PortalHealthSource
Date:          17/11/2009 11:23:50 a.m.
Event ID:      10
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      hostname.domain
Description:
The Portal cannot connect to the middle tier using the web service interface.  This failure prevents all portal scenarios from functioning correctly.

The cause may be due to a missing or invalid server url, a downed server, or an invalid server firewall configuration

In my case I entered a wrong service URL during the installation. I configured the following options to get a real detailed error in IE instead of the “service not available” error. To achieve this alter the following parameters in the web.config of the FIM Portal Sharepoint Site (default: C:\inetpub\www\wss\virtualdirectory\80\web.config)

<SafeMode MaxControls="200" CallStack="True" DirectFileDependencies="10" TotalFileDependencies="50" AllowPageLevelTrace="false">

<customErrors mode="Off" />

<httpModules>

<clear />

<!-- <add name="ILMError" type="Microsoft.IdentityManagement.WebUI.Controls.ErrorHandlingModule, Microsoft.IdentityManagement.WebUI.Controls, Version=4.0.3531.2, Culture=neutral, PublicKeyToken=31bf3856ad364e35" /> -->

<add name="…

…

</httpModules>

In words: set CallStack to True, switch customErrors Off and make sure to comment out the “ILMError” part.

After enabling this logging and doing an IISRESET, visiting the portal gave the following error below. There I could clearly see that the URL being used was wrong.

My source: Anthony Ho on the TechNet Forums: Register user to complete self-service password reset

An alternative, which I haven’t tested yet is enabling the server side tracing for the FIM Service in the Microsoft.ResourceManagement.Service.exe.config file which can be found in C:\Program Files\Microsoft Forefront Identity Manager\2010\Service\, follow the following steps:

1. Uncomment the 3 sections in the config file as explained in the config file itself
2. change "fimDiagnostics.svclog" to "C:\Logs\fimDiagnostics.svclog"
3. Update Error to "Verbose" (capital V) in <source name="Microsoft.ResourceManagement" switchValue="Error">
4. Create the folder C:\Logs
5. restart FIMService

My source is Anthony Ho again on the TechNet Forums: FIM - "An error occured when attempting to reset your password, please try again”

When configuring an Active Directory Management Agent (AD MA) in FIM, a service account has to be foreseen which will be used to connect to Active Directory and perform changes. Most of the guides kindly suggest to use “administrator”, or at least they use it as an example.

However If you want to use an account with limited (read: normal) privileges it’s actually quit easy to setup:

• Create/Delete/Modify permissions for user objects in the OU’s where FIM will maintain user objects.
• “Replicating Directory Changes” on the directory partition of the domain you’re connecting FIM to

If you forget to configure this last option, you will see the following error:failed-search replication access was denied. Error Code 8453

As a reference the following KB article can be used: KB303972: How to grant the "Replicating Directory Changes" permission for the Microsoft Metadirectory Services ADMA service account

In the article two possible procedures are explained: using ADUC or Adsiedit. If you want to perform this change using the command line, the following command can be used:

• dsacls dc=corp,dc=contoso,dc=com" /G CORP\FIM_AD_Permisssions:CA;"Replicating Directory Changes"

Also do not forget to run an import with the AD MA first before trying to provision users with the declarative provisioning from the portal. If you forget to do this you will get the error Object "CN=user,OU=containsusers,DC=corp,DC=contoso,DC=com" does not have a parent object in management agent "AD MA"."

This second Kerberos Basic Troubleshooting post will try to provide some tips and knowledge when setting up Kerberos authentication for SQL services. For starters I will explain which Service Principal Names (SPN’s) can be registered and when they will be called for.

There are three known forms of SPN’s which can be registered for a SQL service account. This service account can either be a user (service account) or a computer (local system) (MSDN: Service Principal Name (SPN) Support in Client Connections)

• MSSQLSvc/FQDN
• MSSQLSvc/FQDN:InstanceName
• MSSQLSvc/FQDN:Port

And here is where the confusion starts: depending on the protocol used to to connect to SQL, other SPN’s are built by the client and being queried to AD. In my line of work I hardly ever encounter the use of “named pipes” or “shared memory”. In 100% of the cases (mostly SQL services for Infrastructure Apps) I see TCP/IP being used. Although I provided 3 options for SPN’s, only one will be used for TCP/IP:

• MSSQLSvc/FQDN:Port

This is also explained in the MSDN: Registering a Service Principal Name article:

User-specified SPNs are supported in client drivers. However, if an SPN is not provided, it will be generated automatically based on the type of a client connection. For a TCP connection, an SPN in the format MSSQLSvc/FQDN:[port] is used for both the named and default instances.

I would just forget the other formats, unless you are using named pipes/shared memory.

Whenever you are using a dynamic port you got a problem. Because each time SQL Services get started, the TPC/IP port can change and hence the SPN can be incorrect. There are two options for this problem, in my opinion the second option is the preferred one:

• Alter the permissions on the service account so it can update it’s own SPNs: grant “SELF” the “Write servicePrincipalName” permission.

• Configure a static port for the SQL Service. A static port can still be combined with the SQL Browser Service. So for applications which do not support entering a fixed SQL Port should be fine as well. An example is shown below: the FIM Portal & Service setup asks for a server and optional Instance, but no port is asked for. You could fool the setup perhaps by providing it anyway using a comma or double point. But having a static port configured and the browser service started will do just fine.

Although I just said to forget the other SPN’s, you might actually use them in some cases. The new SQL 2008 native client has a new feature which allows you to actually configure which SPN the client will query for during authentication. So the client it self will no longer just build the SPN by using FQDN:Port. In this case I does not matter which port SQL is using, you can simply register the SPN MSSQLSvc/FQDN:InstanceName (or something else you like) as long as you provide it in the ODBC connection settings.

I’m quit sure if the ODBC configuration allows this feature It won’t take long for the application installers to allow it. It would be nice to provide an optional SPN when configuring a SQL connection during a regular application setup. This would definitely provide more flexibility.

And now some diagnostic stuff. So you got your new shiny SQL configured, you registered the SPN and installed your application. But who can tell whether your using Kerberos or not? Here are some options:

• Wireshark: the wire never lies. However this requires you to install Wireshark, capture a trace, …
• Security event log: It’s all in there, however there is a lot in there…
• SQL query: It’s just easy!

Start the SQL Management Studio, select the SQL Server in the left pane and click New Query

Provide the following code, select it and press F5 (execute)

The code itself:

SELECT DB_NAME(dbid) AS DatabaseName, loginame AS LoginName, sys.dm_exec_connections.auth_scheme as AuthMethod
FROM sys.sysprocesses
JOIN sys.dm_exec_connections
ON sys.sysprocesses.spid=sys.dm_exec_connections.session_id
WHERE dbid > 0
GROUP BY dbid, loginame, spid,sys.dm_exec_connections.auth_scheme

And as a result the following information is provided: you can clearly see some of the connection are NTLM and some of them are Kerberos. The connections to the master/msdb being NTLM is just because those are connections on the machine itself. Remote connections will properly use Kerberos when the required SPN’s are registered.

Another reference on the topic: MSDN: How to Implement Kerberos Constrained Delegation with SQL Server 2008

A colleague of mine pointed me the following facts:

• Windows 2008 RTM does not support Windows 7 as KMS clients
• Windows 2008 RTM + SP2 + KB968912 does support Windows 7 as KMS clients
• Windows 2008 (in any version) does not support Office 2010 KMS clients
• Windows 2003 does support Office 2010 KMS client (with the Microsoft Office 2010 KMS Host License Pack)
• Windows 2008 R2 supports both (probably the Microsoft Office 2010 KMS Host License Pack is required as well)

So there are two options: Windows 2003 based KMS or Windows 2008 R2 based KMS.

Reference: Microsoft Volume Licensing: Product Activation

It seems like there is some graphical tool to manage or check KMS statistics: “Volume Activation Management Tool (VAMT) 2.0”, I’ll definitely try this one out if I find some time.

A screenshot from VAMT from the Manage Activation Using VAMT 2.0_White Paper 

[update 9/06/2010] The following URL is a reference for KMS related stuff: Volume Activation Deployment Guide

Active Directory Migration Toolkit 3.1 does not install on Windows 2008 R2, however there is nothing wrong with installing it on Windows 2008 x64. When you start playing around with the ADMT vbscripting options you might run into the following error: “ActiveX component can’t create object: ‘ADMT.Migration"’.

The solution is easy, run the script with the x86 version of cscript.exe: “c:\windows\syswow64\cscript.exe script.vbs”

My source: TechNet forums

Why was I toying around with the vbscript instead of using the admt.exe commandline options? I wanted a way to find out which attributes are excluded by ADMT. It seems that one the first run ADMT gathers a list of attributes it will always exclude for migration. Examples are “mail” or “proxyAddresses”. The ADMT help states this could only be done through scripting. This is explained in detail in KB937537.

When configuring a GAL Management Agent (MA) in FIM 2010, you can make sure you have some additional logging by enabling this functionality in the logging.xml configuration file. This can be particularly useful when you want to extend or alter the code and want to add some custom logging. the file can be found at “C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\Extensions” and is called logging.xml. Just change the logging-level from 0 to 1.

The output is either stored in “C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData” or “C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service\MaData\MAfolder” and a sample is shown below:

I intended to write a single entry about Kerberos Basic Troubleshooting, but it would have grown way to long. So instead I decided to write one tip a week. The tips should help you out with simple Kerberos issues. Keep in mind that anyone can run into Kerberos issues whilst not specifically enabling a service for Kerberos. Since Windows 2000 Kerberos is the default authentication mechanism and as such can cause trouble to you. So perhaps I should say “Authentication Basic Troubleshooting tips”.

The tip of today will be KLIST (and Kerbtray).

Klist can list or purge tickets for a given session and is available in W2008 and up. For W2003 or lower klist is available in the resource kit tools. Some sample usage scenario’s:

Usage 1: “klist”: list the tickets of the current user . You can clearly see the difference between the “Ticket Granting Tickets” and the “Service Tickets”. Mostly the Service Tickets are the ones of interest.

Pitfall: you have to run klist from a non UAC elevated prompt. If you run klist in an UAC elevated prompt, you will get a list of tickets your user has inside that specific session. When doing a “run as administrator” for the cmd prompt, a new logon session is made.

Usage 2:”klist purge”: throw away all tickets of the current user

Usage 3: “klist –li 0x3e7” and “klist –li 0x3e7 purge”: allows you to list the tickets of a logon session specified as 0x3e7.

Why is this so special? On each machine 0x3e7 is the session of the machine (“Local System”) itself. How can this be interesting? Use the purge option and you have a way to refresh the group membership of a machine without having to reboot it! Which in turn can be useful for refreshing group policies based on group security filtering. Some references for this topic: topic at activedir.org and Picking up Computer Group Membership Changes without a Reboot.

An example of some tickets of a machine:

As an alternative Kerbtray can be used which is graphical. However it can only be used to show the tickets of the current logged in user. It is available through the Windows 2003 resource kit tools. I like this one on an USB stick when troubleshooting Kerberos issues on Windows XP workstations. Here you can see it on a Windows 2008 R2 server, still works without issues.

You can right click it and choose “List Tickets” or “Purge Tickets”. List Tickets show you the following:

So that’s it for today, next week another Tip will be posed.

Microsoft finally released a System Center Operations Manager management pack for FIM 2010. The MP is still in pre release though.

For those willing to test the MP: it can be downloaded from Microsoft Connect: FIM 2010 Pre-release Management Pack

If you install the MSI you will get an EULA, readme, a guide and of course a MP. In the guide an overview is given of what is monitored:

• FIM Service
• FIM Synchronization Service
• FIM Portal

Client components or components on which FIM depends, such as AD, IIS, SQL or not included in this MP.