The other day one of my customers mentioned that their SCOM Web Portal has been broken for a while now. As I like digging into web application issues I took a quick look. Here’s what I came up with. It seems that the portal itself was loading fine, but viewing All Alerts or Active Alerts showed a Service Unavailable (“HTTP Error 503: The service is unavailable”).

One of the things about IIS based errors is that in most cases the Event Log on the web server can help you a great deal. In the System Event Log I found the following:

A process serving application pool 'OperationsManagerMonitoringView' reported a failure trying to read configuration during startup. The process id was '6352'. Please check the Application Event Log for further event messages logged by the worker process on the specific error. The data field contains the error number.

Checking the IIS Management Console I could indeed see that the Application Pool was stopped. Starting it succeeded, but viewing the page made it crash again. Looking a bit further I found the following in the Application Event Log:

The worker process for application pool 'OperationsManagerMonitoringView' encountered an error 'Configuration file is not well-formed XML

' trying to read configuration data from file '\\?\C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\web.config', line number '14'. The data field contains the error code.

Now that seems pretty descriptive! Using notepad I checked the contents of the file and tried to see why the XML was not well-formed. I checked the XML tags and the closings and such but I couldn’t find anything at first sight. Looking a bit longer I saw that the quotes (“) were different from the other quotes in the file. Here’s a screenshot of the bad line and the fixed line. You can simply erase and retype the “ on that line and you should be good to go.

Personally I like taking a backup copy before I perform manual fixes. After saving the file I did an IISReset just to be sure. And after that we were able to successfully view our alerts through the Web Portal again!

Recently I got a request of a customer to update the root certificates of several certificates they had in place. The problem was that one of the intermediate CA’s had an expiration date which was before the expiration date of the actual certificate. Here’s the information we got with this notification.

The problem was the Belgium Root CA2. It’s valid until 27/01/2014 whilst several of the “your servercertificate (SSL)” are valid till the end of 2014. When clients would validate this chain after the 27th of January this would cause problems. With this news we received the new root and intermediate CA’s in a zip file.

Using certutil you can easily install them in the required stores on the server which has “your servercertificate (SSL)” configured for one or more services.

• certutil -addstore -enterprise Root "C:\Temp\NewRootChain\Baltimore Cybertrust Root.crt"
• certutil -addstore -enterprise CA "C:\Temp\NewRootChain\Cybertrust Global Root.crt"
• certutil -addstore -enterprise CA "C:\Temp\NewRootChain\Belgium ROOT CA 2.crt"
• certutil -addstore -enterprise CA "C:\Temp\NewRootChain\government2011.cer"

After performing these steps I could see the new chain reflected in my certificate on the server. Now I figured that the clients should retrieve this chain as well one way or another. Upon accessing https://web.contoso.com I could see that the certificate was trusted, but the path was still showing the old chain!

First thing I verified was that the “Baltimore Cybertrust Root” was in the trusted root certificate authorities of my client. Without me actually putting it there it was present. This makes sense as this probably comes with windows update or something alike. I assume the client has to retrieve the intermediate certificates himself. I thought that he would go externally for that. From the certificate I found the Authority Information Access URL which pointed to the (outdated) Belgium Root CA2 on an external (publicly available) URL. “AHAH” I figured, time to contact the issuers of these certificates. They kindly replied that if the server has the correct chain, the clients should reflect this. They also provided me an openssl command and requested more information.

This made me dig deeper. After a while I came to the following conclusion: my “bad” client showed different paths for these  kind of certificates… When visiting my ADFS service I saw the correct chain being build, but on my web server I had the old chain. Very odd. So something had to be wrong server side. From what I can tell here’s my conclusion:

The browser gets the intermediate certificates in the chain from the IIS server

• IIS 8.0 on Windows 2012: update the stores and all is good (or the servers had a reboot somewhere in the last weeks that I’m unaware off)
• IIS 7.5 on Windows 2008 R2: update the stores AND unbind/bind the certificate in the IIS bindings of your website(s).

For the IIS 7.5 I also tried an IIS reset, but that wasn’t enough. Perhaps a reboot would do too.  Here’s my source for the solution: http://serverfault.com/questions/238206/iis7-not-sending-intermediate-ssl-certificate

A usefull openssl command, even works for services like sldap. It will show you all certificates in the chain.

• openssl.exe s_client -showcerts -connect contoso.com:636
• openssl.exe s_client -showcerts -connect web.contoso.com:443

P.S. The new chain also has an oddity… Belgium Root CA2 is valid until 2025 whilst the Cybertrust Global Root expired 2020

Bonus tip #1: in the windows event log (of the client) you can enable CAPI2 logging. This will show you detailed information of all Certificate related operations. In my opinion the logging is often to detailed to tell you much, but it’s nice to know it’s there. You can find it under Application and Services\Microsoft\Windows\CAPI2 right-click Operational and choose enable.

Bonus tip #2: on Windows 2012/ Windows 8 you can easily open the certificates of both the current user and the current computer. In the past I often used mmc > add remove certificates > click some more > … Now there’s a way to open a certificates mmc for the local computer using the command line:

• Current user: certmgr
• Current computer: certlm

[Update 20/11/2013] forgot to mention the -enterprise switch in the certutil commands. This ensures the local computer certificate stores are used.

Tip #1 (IIS): appcmd and IIS bindings:

Some more IIS (configuration) awesomeness: you can easily view the bindings for an IIS site using the following command:

• appcmd list site /site.name:”MySite”

Now obviously just viewing isn’t that cool, but you can also set them! This is extremely useful for those environment where you have work according to “Development –> Test –> Acceptance –> Production” or other variances. I hate doing the same task (manually) multiple times.

So here’s how you can “push” your bindings to a site called “MySite” in IIS.

Syntax option #1: just a host header available on all IP’s (*):

• appcmd set site /site.name:”MySite” /bindings:”http://mysite.contoso.com:80”,”http://mysite:80”

Syntax option #2: a host header bound to one specific IP:

• appcmd set site /site.name “MySiite” /bindings:”http/192.168.50.11:80:mysite.contoso.com”,”http/192.168.50.11:80:mysite”

Mind the difference in the bindings parameters syntax: e.g. http:// <> http/

Now what if you want to change one specific binding to an other value?

• appcmd set site /site.name:"MySite" /bindings.[protocol='http',bindingInformation=':80:mysite.contoso.com'].bindingInformation:192.168.50.11:80:mysite.contoso.com

In this example I changed the binding which was listening on all IP address to only listen on a specific IP address.

Or what about adding a binding witouth modifying existing bindings?

• appcmd set site /site.name:"MySite" /"+bindings.[protocol='https',bindingInformation='192.169.16.105:443:mysite.contoso.com’]

P.S. appcmd is an executable which you can find in the c:\windows\system32\inetsrv directory.

Tip #2 (SQL), is SQL Full Text Search installed?:

One of the prerequisites when installing the FIM Service is that the SQL Full Text Search feature is installed on the SQL Instance hosting your database. There’s two easy ways to see if this is the case:

• Using the services.msc mmc: check if there’s a service name SQL Server FullText Search ([instance]) where [instance] will be the name of your instance
• Using the following SQL query: IF (1 = FULLTEXTSERVICEPROPERTY('IsFullTextInstalled')) print 'INSTALLED' else print 'NOT INSTALLED'

My source: serverfault.com: How to detect if full text search is installed in SQL Server

Tip #3 (Visual Studio): auto-increment version info:

When you create a class in C# or your preferred language you might want to have some version information on the DLL you build. There’s an easy way to configure your solution/project to auto-increment the build version every time you compile your project.

You can do this by directly editing the AssemblyInfo.cs below the properties.

// Version information for an assembly consists of the following four values:
//
//      Major Version
//      Minor Version
//      Build Number
//      Revision
//
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.0.*")]
//[assembly: AssemblyFileVersion("1.0.0.0")]

In words: make sure to set the AssemblyVersion to something like “1.0.*” or “1.0.0.*” and comment the AssemblyFileVersion line. As I tested around a bit I noticed the following things:

• the AssemblyFileVersion does not work with the *
• If the AssemblyFileVersion is not commented the AssemblyVersion is ignored and the AssemblyFileVersion “wins”.

Some more background information:

• Marc Charbonneau’s Blog: Auto-incrementing build numbers in Visual Studio
• stackoverflow.com: How to have an auto incrementing version number (Visual Studio)?
• MSDN: AssemblyVersionAttribute Class

Tip #4 (Certificate Authority): RPC traffic check

Often when playing around with certificates I’m hitting gpupdate like hell in order to retrieve auto-enrollment. But if you want to make sure your CA is actually reachable from a given endpoint over RPC/DCOM you can easily check this using the certutil utility. This utility is available out of the box.

• certutil -ping -config "ca-server-name\ca-name”
• Example: certutil –ping –config “SRVCA01\Customer Root CA”

I’m a bit stumbled that I’ve only come across this now. Recently I discovered that there are some cases where you can end up with your service account using a temporary profile. Typically this is the case where your service account has very limited privileges on a Server. Like application pool identities which run as a regular AD user, which I consider a best practice. I myself saw this in the context of the application pool identities in a SharePoint 2010 farm or with SQL Server Reporting Services 2008 R2.

The phenomena is also described at: Todd Carter: Give your Application Pool Accounts A Profile So this does not apply to all Application Pool identities! Only those running with “load profile=true”.

In the Application event log you can find the following event:

Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.

How to fix it if you see those nasty “c:\users\TEMP” folders?

1. Stop the relevant application pools
2. Stop the IIS Admin Service (in services.msc)
3. See that the TEMP folders are gone in c:\users
4. Follow the next steps

How to make sure your accounts get a decent profile?

We will temporary add the service account to the local administrators group so they can create a profile. In fact all they need is the “logon locally” privilege. The second command will start a command prompt while loading a profile. This will ensure a proper profile is created.

1. net localgroup administrators CONTOSO\AppPoolAccount /add
2. runas /u:CONTOSO\AppPoolAccount /profile cmd
3. net localgroup administrators CONTOSO\AppPoolAccount /del

As a side note: if the TEMP folders are not disappearing, or you are still getting a temporary profile, you can try to properly cleanup the temporary profile:

1. Stop the application pools
2. Stop the IIS Admin Service
3. Using right-click properties on computer, choose advanced tab and then pick User Profiles. There you can properly delete them.

If you’re still having troubles you might need to delete the TEMP folders manually AND cleanup the following registry location: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList Especially look if there aren’t any keys with .bak appended to it.

Ever modified something in IIS and then wanted to revert but forgot what setting it actually was? Or ever been in a lot of trouble, you know somebody changed something, but not what? Well seems IIS keeps backups of the configuration files for you!

You can find them in c:\inetpub\history

I don’t think they’ll be created when you manually modify those xml configuration files. Might be interesting to check if performing modifications through appcmd make it here. I would guess so.

Just a quick tip: often after an IIS or TMG project security audits are performed. One of the remarks we often get is that SSL 2.0 is enabled on the web server. To disable this you can simply set the required reg key as explained in KB187498

One of the typical things of a webapp deployed on IIS is that everyday the first visitor has a very slow loading page. This is due to the application pool recycling. Every night, somewhere between 2 and 4 (for default installs) the application pools are recycled. After this when the first visitor arrives certain code is loaded again which takes some time. Consequent visitors don’t suffer this phenomena.

In the IIS world there’s a known solution for this: "IIS Warm Up scripts”. There even was  a built-in module for this in IIS 7.5, however it was in beta state and currently has been removed from the web. With some PowerShell and Scheduled Task magic we can warm up IIS just as good. In my example the first user was experiencing a 15-20 seconds delay before the FIM Portal showed up. After applying the following solution that was more like 2 seconds.

Our script which will warm up the site will do this every time the application pool is recycled. The trigger for the scheduled task will be an event in the Application event log. Therefor we have to modify the default settings of the application pool so that it logs these events.

Open the IIS management console and locate the application pool responsible for the SharePoint site hosting your FIM portal

Righ-click and choose advanced settings, make sure to set “Specific Time” to True.

Every time the application pool is recycled by the schedule defined in the “Specific Times” array the following even will be logged:

A worker process with process id of '4420' serving application pool 'SharePoint - 80' has requested a recycle because it reached its scheduled recycle time.

Now the following script can be executed to warm up IIS. It will do this by visiting the page and performing a get. It’s very basic and probably can be improved or rewritten.

function get-webpageWithAuthN([string]$url,[System.Net.NetworkCredential]$cred=$null){
    write-host -foregroundcolor green "Warming up $url";
    $wc = new-object net.webclient;
    $wc.credentials = $cred;
    #$wc.Headers.Add("user-agent", "PowerShell");
    $html = $wc.DownloadString($url);
    #$html
}

#FIM
$website = http://FIM.setspn.blogspot.com/IdentityManagement
$credentials = [System.Net.CredentialCache]::DefaultCredentials;
get-webpageWithAuthN -url $website -cred $credentials

Now we can create the Scheduled Task. I choose to run it as “local system”. This seems enough to warm up the portal. I think because this way the FIM Service is queried to see if “Local System” (my FIM server itself) is known in the portal. Obviously this is not the case and I will probably see a “Service not available” error. But the point is that the Portal is warmed up!

We will add some triggers:

• Type: On an event
• Log: System
• Source: WAS
• Resource ID: 5076

Then we will specify an action:

• Action: Start a program
• Program/Script: PowerShell.exe
• Add arguments: c:\yourfolder\script.ps1
• Start in: c:\yourfolder

If we also want to trigger the script when we an administrator recycles an Application Pool we have to add a trigger for event 5079 (System –> WAS: 5079)

If we also want to trigger the script when we perform an IISreset we can add a trigger for the following event 3201 (System > IIS-IISREeset: 3201)

Obviously this way of working could be applied for other websites too.

In the past year I’ve become more and more interested and familiar with Kerberos authentication. While I’m not saying that you should “Kerberize” everything, I think everyone installing and configuring apps on the Windows platform should have a basic understanding of it.

Below is a decision-based workflow I created to counter some simple pitfalls. Although some of it might seem easy, it gets forgotten a lot. In the example a user is browsing a web-based application which is reachable at “webapp.contoso.com”. In fact this website is hosted on a server called web01.contoso.com.

Important to note that ending up in the orange field (“client uses NTLM”) isn’t necessarily bad, but it might be when your web app does some form of delegation afterwards. On the other hand, if you end up in the “authentication impossible”, you will never-ever get granted access to the application.

This example is based on a web-based application, but the reasoning is exactly the same when the IE browser is a SQL client and the application pool for the website is a SQL Server service.

Perhaps the most common one to be encountered is the one where someone uses a service account for an application pool instead of the network service. If you then try to access the website with the name of the machine, you will always end up in the “authentication impossible”.

Any feedback or comments is highly appreciated. The chart, click the picture for a clearer view: