One of my customers had problems with certain users being member of a lot of Active Directory groups. This resulted in several client side issues. There’s an easy and well-known “fix” for that: raise the MaxTokenSize registry key on all Windows operating systems in your domain. On Windows 8(.1) / 2012 (R2) the MaxTokenSize is already at its maximum (advised) value out of the box. That value is 48.000 bytes. In order to mitigate these users their access problems we raised the MaxTokenSize to 48.000 bytes on all clients and servers that are running Windows 7/ Windows 2008 R2. After this change the typical issues were gone. However new ones came up:

From time to time, when HTTP is involved, issues were encountered:

• Opening the Direct Access management console (depends on WinRM)
• Open the FIM Portal
• Streaming App-V packages over HTTP
• …

Typically the user would receive several authentication prompts and even after specifying valid credentials another prompt would reappear. Example browser based issue:

As you can see the browser gives an HTTP 400 Bad Request error. Using a network trace we can easily see why it’s considered bad:

And the packet details:

The details clearly state that The size of the request headers is too long.

The problem here is that the token is allowed to be up to 48.000 bytes where it used to be 12.000 bytes. The http subsystem of a windows server has several parameters that are supposed to protect the server from oversized requests. However, as the token can now be a lot larger, the maximum request size has to be tuned as well:

From: KB820129

Below: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters there are two interesting values:

And from: KB2020943 we can find a formula to calculate the MaxFieldLength to set based on the MaxTokenSIze.

If MaxToken is 48.000 bytes (default in Windows 2012 and configure by GPO for 2008 R2/ Win7):

• (4/3 * 48000) + 200 = 64200

We’ll use the maximum allowed value of MaxFieldLength 65534 (=~ 64200) to allow tokens up to 48000 bytes. We’ll also use this value for MaxRequestBytes.

• MaxFieldLength: we can take the maximum allowed value: 65534
• MaxRequestBytes:  65534

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters

Other useful information:

• http://blogs.msdn.com/b/ashishsingh/archive/2010/04/03/windows-authentication-for-accounts-with-large-kerberos-tickets-may-not-work-despite-having-maxtokensize-in-place.aspx
• http://www.grouppolicy.biz/2013/06/how-to-configure-iis-to-support-large-ad-token-with-group-policy/

I specifically wanted to post this information as in many other only articles/posts I always see people just using the maximum allowed value for MaxRequestBytes and I don’t feel 100% comfortable with that. Second,  in my opinion it’s advised to have these values pushed out to all your server systems. Especially now that Windows 2012 and up have a MaxTokenSize of 48.000 by default. If you don’t push these HTTP.sys parameters, you’ll end up troubleshooting the same phenomena multiple times from different angles. Why waste time?

Recently I helped a customer which was having troubles migrating from a Windows 2003 cluster to a Windows 2012 cluster. The resources they were running on the cluster consisted of many in house developed applications. There were about 80 of them and they were running as generic applications.

Due to Windows 2003 being end of life they started a phased migration towards Windows 2012 (in a test environment). At first the migration seemed to go smooth, but at a given moment they were only able to start a limited amount of applications. The applications that failed gave an Out Of Memory exception (OutOfMemoryException). Typically they could start about 25 applications, and from then on they weren’t able to start more. This number wasn’t exact, sometimes it was more, sometimes it was less.

As I suspected that this wasn’t really a failover clustering problem but more a Windows problem I googled for “windows 2012 running many applications out of memory exception”. I found several links:

HP: Unable to Create More than 140 Generic Application Cluster Resources

IBM: Configuring the Windows registry: Increasing the noninteractive desktop heap size

If the parallel engine is installed on a computer that runs Microsoft Windows Server, Standard or Enterprise edition, increase the noninteractive desktop heap size to ensure that a sufficient number of processes can be created and run concurrently

So it seems you can tweak the desktop heap size in the registry. Here is some background regarding the modification we did to the registry.

The key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

SharedSection has three values: (KB184802: User32.dll or Kernel32.dll fails to initialize)

• The first SharedSection value (1024) is the shared heap size common to all desktops. This includes the global handle table, which holds handles to windows, menus, icons, cursors, and so forth, and shared system settings. It is unlikely that you would ever need to change this value.
• The second SharedSection value (3072) is the size of the desktop heap for each desktop that is associated with the "interactive" window station WinSta0. User objects like hooks, menus, strings, and windows consume memory in this desktop heap. It is unlikely that you would ever need to change this second SharedSection value.
• The third SharedSection value (512) is the size of the desktop heap for each desktop that is associated with a "noninteractive" window station. If this value is not present, the size of the desktop heap for noninteractive window stations will be same as the size specified for interactive window stations (the second SharedSection value).

Default on Windows 2012 seems to be 768

Raising it to 2048 seems to be a workaround/solution. A reboot is required! After this we were able to start up to 200 generic applications (we didn’t test more). However after a while there were some failures, but at first sight quite limited. This might be due to the actual memory being exhausted. Either way, we definitely saw a huge improvement.

Disclaimer: ASKPERF: Sessions, desktops and windows stations

Please do not modify these values on a whim. Changing the second or third value too high can put you in a no-boot situation due to the kernel not being able to allocate memory properly to even get Session 0 set up

Bonus info: why didn’t the customer didn’t have any issues running the same workload on Windows 2003? They configured the generic applications with “allow desktop interaction”. Something which was removed from the generic applications in Windows 2008. Because they had “allow desktop interaction” configured, generic applications were running in an interaction session and thus were not limited by the much smaller non-interactive desktop heap size.

Recently I worked together with some Microsoft consultants to implement a Direct Access proof of concept. Initially the configuration went just fine. However, the operational state page showed that we had an issue with the IPSEC certificate. Initially we thought the problem had to do with our DA server being unable to reach the internet in order to download the CRL (Certificate Revocation List). So we had two options: either ask the network team to open up the DA’s external interface connectivity towards the internet or configure the DA server so it could use a proxy.

Due to the workload of the network team, not the best reasoning though, we choose option two: the proxy. We configured the proxy in IE and we were able to successfully download the CRL file. The error didn’t go away though. So we figured we had to configure the system to use the proxy. This can be done using netsh:

• netsh winhttp set proxy 10.0.10.30:8080

The error didn’t went away and it was time for lunch so we thought we’d give the DA server a reboot and some time. When we came back there was something weird with the operation state page. It just said Name_Of_DA_Server is not working properly. We tried rerunning the configuration wizard but we got greeted with the following error:

In words: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. Suddenly we figured out that our netsh winhttp proxy might have broken things a bit. So we gave the command a little twist:

• netsh winhttp set proxy 10.0.10.30:8080 bypass-list="<local>;*.contoso.local"

Now that we used this command our DA console was back to normal, but our IPSEC certificate was still in error. The solution was actually quite simple: just make sure the DA server also has a certificate based upon the Computer (or your custom computer) template. Such a certificate can be used for both client and server authentication. This certificate should be issued by the same CA that issues certificates for your clients and that is configured in the DA configuration wizard. After enrolling a certificate we now had a working DA setup. We could see our clients setting up a connection successfully.

One thing that bothered us though: on the operation state page we had an error on the availability of the IPHTTPS interface of the DA server. Now that was odd. netsh showed it as available and after all we had several clients connected to it. When we removed the proxy (netsh winhttp reset proxy) and then the status page went all green!

So for now I’m not even sure if the DA server needs to check the CRL’s of the certificate used by IPHTTPS. But I guess we’ll find out soon if DA stops working. All I can say: the netsh winhttp set proxy doesn’t seems a really good idea to use. Things started breaking in weird and unpredictable ways.

I’m not sure whether there will be another person ever running into this, but if you do I hope this saves you some time.

We’re currently defining a new build for our file servers. On one of the servers we installed in the test environment we started seeing a lot of errors in the Application event log.

The events we were seeing:

Event 12344:

File Server Resource Manager finished syncing claims from Active Directory and encountered errors during the sync (0x80072030, There is no such object on the server.
).  Please check previous event logs for details.

Event 12339:

File Server Resource Manager failed to find the claim list 'Global Resource Property List' in Active Directory (ADsPath: LDAP://domaincontroller.contoso.com/CN=Global Resource Property List,CN=Resource Property Lists,CN=Claims Configuration,CN=Services,CN=Configuration,DC=contoso,DC=com). Please check that the claims list configured for this machine in Group Policy exists in Active Directory.

As you can see these were piling up real fast:

From what I can tell these started happing after we configured File Quota’s. In order to do this we added the File Server Resource Manager feature. A quick google led me to the following solution: in order to avoid this errors, a schema upgrade to Windows 2012 is required. Our domain is currently on 2008 R2. I didn’t performed the upgrade just yet, but I wanted to share this nonetheless.

My sources for this information:

• HP Support: Title: Microsoft Windows Server 2012 - Logs Error: "File Server Resource Manager Failed to Find the Claim List Global Resource Property List"
• bsirotin blog: Error 12339 and 12344 on Windows 2012 servers