One of my customers had problems with certain users being member of a lot of Active Directory groups. This resulted in several client side issues. There’s an easy and well-known “fix” for that: raise the MaxTokenSize registry key on all Windows operating systems in your domain. On Windows 8(.1) / 2012 (R2) the MaxTokenSize is already at its maximum (advised) value out of the box. That value is 48.000 bytes. In order to mitigate these users their access problems we raised the MaxTokenSize to 48.000 bytes on all clients and servers that are running Windows 7/ Windows 2008 R2. After this change the typical issues were gone. However new ones came up:

From time to time, when HTTP is involved, issues were encountered:

• Opening the Direct Access management console (depends on WinRM)
• Open the FIM Portal
• Streaming App-V packages over HTTP
• …

Typically the user would receive several authentication prompts and even after specifying valid credentials another prompt would reappear. Example browser based issue:

As you can see the browser gives an HTTP 400 Bad Request error. Using a network trace we can easily see why it’s considered bad:

And the packet details:

The details clearly state that The size of the request headers is too long.

The problem here is that the token is allowed to be up to 48.000 bytes where it used to be 12.000 bytes. The http subsystem of a windows server has several parameters that are supposed to protect the server from oversized requests. However, as the token can now be a lot larger, the maximum request size has to be tuned as well:

From: KB820129

Below: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters there are two interesting values:

And from: KB2020943 we can find a formula to calculate the MaxFieldLength to set based on the MaxTokenSIze.

If MaxToken is 48.000 bytes (default in Windows 2012 and configure by GPO for 2008 R2/ Win7):

• (4/3 * 48000) + 200 = 64200

We’ll use the maximum allowed value of MaxFieldLength 65534 (=~ 64200) to allow tokens up to 48000 bytes. We’ll also use this value for MaxRequestBytes.

• MaxFieldLength: we can take the maximum allowed value: 65534
• MaxRequestBytes:  65534

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters

Other useful information:

• http://blogs.msdn.com/b/ashishsingh/archive/2010/04/03/windows-authentication-for-accounts-with-large-kerberos-tickets-may-not-work-despite-having-maxtokensize-in-place.aspx
• http://www.grouppolicy.biz/2013/06/how-to-configure-iis-to-support-large-ad-token-with-group-policy/

I specifically wanted to post this information as in many other only articles/posts I always see people just using the maximum allowed value for MaxRequestBytes and I don’t feel 100% comfortable with that. Second,  in my opinion it’s advised to have these values pushed out to all your server systems. Especially now that Windows 2012 and up have a MaxTokenSize of 48.000 by default. If you don’t push these HTTP.sys parameters, you’ll end up troubleshooting the same phenomena multiple times from different angles. Why waste time?

Today I got a call from a colleague asking me to assist with an issue. His customer had a Windows 2008 R2 server with a custom .NET application on it. The application seemed to stop working from time to time. After a reboot the application seemed to work for a while.

The logging showed a stack trace that started at UserPrincipals.GetAuthorizationGroups and gave the message: An error (1301) occurred while enumerating the groups. The group's SID could not be resolved.

Exception information:
Exception type: PrincipalOperationException
Exception message: An error (1301) occurred while enumerating the groups. 
The group's SID could not be resolved.
at System.DirectoryServices.AccountManagement.SidList.TranslateSids(String target, IntPtr[] pSids)
at System.DirectoryServices.AccountManagement.SidList..ctor(SID_AND_ATTR[] sidAndAttr)
at System.DirectoryServices.AccountManagement.AuthZSet..ctor(Byte[] userSid, NetCred credentials, ContextOptions contextOptions, String flatUserAuthority, StoreCtx userStoreCtx, Object userCtxBase)
at System.DirectoryServices.AccountManagement.ADStoreCtx.GetGroupsMemberOfAZ(Principal p)
at System.DirectoryServices.AccountManagement.UserPrincipal.GetAuthorizationGroups()

The first thing that came to mind was that they deleted some groups and that the application wasn’t properly handling that. But they assured me that was not the case. The only thing they had changed that came to mind was adding a Windows 2012 domain controller.

I could easily reproduce the issue using PowerShell:

Function Get-UserPrincipal($cName, $cContainer, $userName){
    $dsam = "System.DirectoryServices.AccountManagement"
    $rtn = [reflection.assembly]::LoadWithPartialName($dsam)
    $cType = "domain" #context type
    $iType = "SamAccountName"
    $dsamUserPrincipal = "$dsam.userPrincipal" -as [type]
    $principalContext =
        new-object "$dsam.PrincipalContext"($cType,$cName,$cContainer)
    $dsamUserPrincipal::FindByIdentity($principalContext,$iType,$userName)
}

[string]$userName = "thomas"
[string]$cName = "contoso"
[string]$cContainer = "dc=contoso,dc=local"

$userPrincipal = Get-UserPrincipal -userName $userName '
    -cName $cName -cContainer $cContainer

$userPrincipal.getGroups()

Source: Hey Scripting Guy

Some googling led me to:

• StackOverflow: UserPrincipals.GetAuthorizationGroups An error (1301) occurred while enumerating the groups. After upgrading to Server 2012 Domain Controller
• KB2830145: SID S-1-18-1 and SID S-1-18-2 cannot be mapped on Windows-based computers in a domain environment

In short, it seems that when a 2012 domain controller was involved, the GetAuthorizationGroups() function would choke on two new groups (SIDs) that are added to a user by default. Patching the server running the application was enough in order to fix this.

The issue wasn’t really hard to solve as the solution was easy to find online, but I think it’s a great example of the type of application/code to give a special look when you’re testing your AD upgrade.

Recently I helped a customer which was having troubles migrating from a Windows 2003 cluster to a Windows 2012 cluster. The resources they were running on the cluster consisted of many in house developed applications. There were about 80 of them and they were running as generic applications.

Due to Windows 2003 being end of life they started a phased migration towards Windows 2012 (in a test environment). At first the migration seemed to go smooth, but at a given moment they were only able to start a limited amount of applications. The applications that failed gave an Out Of Memory exception (OutOfMemoryException). Typically they could start about 25 applications, and from then on they weren’t able to start more. This number wasn’t exact, sometimes it was more, sometimes it was less.

As I suspected that this wasn’t really a failover clustering problem but more a Windows problem I googled for “windows 2012 running many applications out of memory exception”. I found several links:

HP: Unable to Create More than 140 Generic Application Cluster Resources

IBM: Configuring the Windows registry: Increasing the noninteractive desktop heap size

If the parallel engine is installed on a computer that runs Microsoft Windows Server, Standard or Enterprise edition, increase the noninteractive desktop heap size to ensure that a sufficient number of processes can be created and run concurrently

So it seems you can tweak the desktop heap size in the registry. Here is some background regarding the modification we did to the registry.

The key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

SharedSection has three values: (KB184802: User32.dll or Kernel32.dll fails to initialize)

• The first SharedSection value (1024) is the shared heap size common to all desktops. This includes the global handle table, which holds handles to windows, menus, icons, cursors, and so forth, and shared system settings. It is unlikely that you would ever need to change this value.
• The second SharedSection value (3072) is the size of the desktop heap for each desktop that is associated with the "interactive" window station WinSta0. User objects like hooks, menus, strings, and windows consume memory in this desktop heap. It is unlikely that you would ever need to change this second SharedSection value.
• The third SharedSection value (512) is the size of the desktop heap for each desktop that is associated with a "noninteractive" window station. If this value is not present, the size of the desktop heap for noninteractive window stations will be same as the size specified for interactive window stations (the second SharedSection value).

Default on Windows 2012 seems to be 768

Raising it to 2048 seems to be a workaround/solution. A reboot is required! After this we were able to start up to 200 generic applications (we didn’t test more). However after a while there were some failures, but at first sight quite limited. This might be due to the actual memory being exhausted. Either way, we definitely saw a huge improvement.

Disclaimer: ASKPERF: Sessions, desktops and windows stations

Please do not modify these values on a whim. Changing the second or third value too high can put you in a no-boot situation due to the kernel not being able to allocate memory properly to even get Session 0 set up

Bonus info: why didn’t the customer didn’t have any issues running the same workload on Windows 2003? They configured the generic applications with “allow desktop interaction”. Something which was removed from the generic applications in Windows 2008. Because they had “allow desktop interaction” configured, generic applications were running in an interaction session and thus were not limited by the much smaller non-interactive desktop heap size.

Recently I had a customer where they had a Certificate Authority in a lab environment and one in a production environment. At first sight both seemed to function correctly. However SCOM (System Center Operations Manager, a monitoring solution) was showing various events of application pool crashes for both environments. The application pool belonged to the CA Web Enrollment pages. When investigating the the Event log on those machines we found the following events to be recurring:

In words:

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: scrdenrl.dll_unloaded, version: 0.0.0.0, time stamp: 0x4a5bc7f2
Exception code: 0xc0000005
Fault offset: 0x000007fef9402594
Faulting process id: 0x10b4
Faulting application start time: 0x01cdee76c8747cfb
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: scrdenrl.dll
Report Id: 795f74be-5a8c-11e2-8b2c-005056ac0079

And also:

In words:

A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '4276'. The process exit code was '0xff'.

The events were recurring, but not very exact. Sometimes it was about every 5 minutes, but sometimes it was once an hour, or even once a day. They only thing we could say that it occurred at least once a day. Besides those events, I couldn’t find anything out of the ordinary on those machines. So off to plan B: google gave me this: TechNet Forums: Prolific number of Windows error reports pertaining to 2008 R2 certificate services.

This seemed to match my problem exactly. I tried the suggestion: removed the CA web components/IIS, reboot, reinstalled. Quickly the events reappeared. The Second thing I noticed here was that SCOM was also involved. Of course, it could be the causing it or the one noticing it….  Disabling the SCOM agent didn’t help. So I started digging deeper. I started looking at the IIS logs. I could see that whilst the CA isn’t visited that regularly, still a lot of requests where logged at frequent intervals. The user agent of the request was mentioning SCOM, so it was pretty obvious this was part of some monitoring configuration.

I asked the guy responsible for SCOM, and besides the regular host based monitoring, they also added URL monitoring (for /certsrv). After disabling this URL monitoring the events stopped occurring. So somehow SCOM doesn’t plays to nice with its requests. I’m not sure why it causes the application pool to crash, maybe it’s something which has to be fixed on the Certificate Authority side, but I’m glad at least I found out WHAT was causing it!

I’m a bit stumbled that I’ve only come across this now. Recently I discovered that there are some cases where you can end up with your service account using a temporary profile. Typically this is the case where your service account has very limited privileges on a Server. Like application pool identities which run as a regular AD user, which I consider a best practice. I myself saw this in the context of the application pool identities in a SharePoint 2010 farm or with SQL Server Reporting Services 2008 R2.

The phenomena is also described at: Todd Carter: Give your Application Pool Accounts A Profile So this does not apply to all Application Pool identities! Only those running with “load profile=true”.

In the Application event log you can find the following event:

Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.

How to fix it if you see those nasty “c:\users\TEMP” folders?

1. Stop the relevant application pools
2. Stop the IIS Admin Service (in services.msc)
3. See that the TEMP folders are gone in c:\users
4. Follow the next steps

How to make sure your accounts get a decent profile?

We will temporary add the service account to the local administrators group so they can create a profile. In fact all they need is the “logon locally” privilege. The second command will start a command prompt while loading a profile. This will ensure a proper profile is created.

1. net localgroup administrators CONTOSO\AppPoolAccount /add
2. runas /u:CONTOSO\AppPoolAccount /profile cmd
3. net localgroup administrators CONTOSO\AppPoolAccount /del

As a side note: if the TEMP folders are not disappearing, or you are still getting a temporary profile, you can try to properly cleanup the temporary profile:

1. Stop the application pools
2. Stop the IIS Admin Service
3. Using right-click properties on computer, choose advanced tab and then pick User Profiles. There you can properly delete them.

If you’re still having troubles you might need to delete the TEMP folders manually AND cleanup the following registry location: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList Especially look if there aren’t any keys with .bak appended to it.

Ok, I’ve gone through my mailbox and I’ve got quite some little neat tricks I want to share and most of all never forget myself. So I’ll put them here for future reference.

Tip #1 (Network):

Remember “Network Tracing Awesomeness” If you’d only want to have traffic captured which involves a specific IP you can start the trace like this:

netsh trace start capture = yes ipv4.address=10.90.101.41

This can be very convenient if your server is a domain controller or a file server and communicates with a lot of clients all the time.

Tip #2 (IIS):

In various IIS Kerberos configuration howto’s you are instructed to set useAppPoolCredentials to true. I Always hate editing XML’s directly as it’s quite easy to make errors. Using the following command you can easily set this parameter from a command prompt:

appcmd set config "Default Web Site" /section:windowsauthentication
/useAppPoolCredentials:true /commit:MACHINE/WEBROOT/APPHOST
(the command is supposed to be on one line)

The Default Web Site is the name of the site as it appears in the IIS management console. Remember, you might need to have something like Default Web Site/vDir If you have to configure this for sublevels of the site.

Tip #3 (Kerberos):

If you enable an account to be trusted for delegation to a given service, you might have to wait some time before the service itself notices this. This is often noticed as: I changed something, it didn’t work and magically the next day it started working. If I’m not mistaken, this might have to do with the Kerberos S4U refresh interval which is at 15’ by default. At least that was the value at Windows 2003… See also: KB824905: Event ID 677 and event ID 673 audit failure messages are repeatedly logged to the Security log of domain controllers that are running Windows 2000 and Windows Server 2003

Tip #4 (PowerShell):

From: MSDN: Win32_PingStatus class

When you use PowerShell to perform remote tasks on a server, such as WMI queries, it might be way more efficient to do a quick ping before actually trying to talk WMI to the server. This way you can circumvent those nasty timeouts when the server you are trying to talk to is down.

$server = "server01"
$PingStatus = Gwmi Win32_PingStatus -Filter "Address = '$Server'" |Select-Object StatusCode

Tip #5(Tools):

Every once in a while I need a tool from the Sysinternals Utilities set. Mostly I go to google, type in the name, get to the Microsoft site hosting the utility and click launch. However, it seems you easily access all of the tools using this webdav share: \\live.sysinternals.com just enter it in a file explorer or your start-> run. The utilities we all know so well are located in the Tools folder. Or if that doesn’t works, just use http://live.sysinternals.com/ 

Thanks to a colleague for this last tip!

-Stay tuned for more!-

One of my colleagues was having troubles accessing an HTTPS site. The site is secured with a certificate coming from an Active Directory Certificate Authority. Now I know of a bug where if you have a pinned website on your taskbar, and from that browser instance you open an HTTPS site with an untrusted certificate, there’s no "continue anyway” button…

Now this wasn’t the case today. He had the “continue anyway” option, where you typically click on, load the site and check the certificate. However, after clicking, it didn’t go trough, it just remained at the same page. We installed the root CA manually in the trusted root authorities, but still no improvements. When verifying the root certificate in the MMC we also saw it mentioned that the digital signature was invalid.. odd!

Using that as a query for google we quickly came across this:

• RSA keys under 1024 bits are blocked
• Blocking RSA Keys less than 1024 bits (part 2)
• How to identify if your ADCS has issued any certificates with public keys <1024 bits (in preparation for KB2661254)

If you read those first two carefully you’ll see the update will be released as a critical non-security update on august 14th for Windows XP, Windows Server 2003, Windows Server 2003 R2, Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2.

An example of a bad certificate:

Now how come he was having this issue now already?! Ahah, now comes the clue! He was using Windows 8! Now I am too, and I’m not having that problem with that specific site, but here’s the difference:

• Windows 8 with issue: Windows 8 Release Preview: build 8400
• Windows 8 without issue: Windows 8 Consumer Preview: build 8250

So it seems they’ve included this update somewhere in the build process of Windows 8.

Having certificates with an RSA key < 1024 is probably not really the case for most of us, but be sure to double check those certificates and their (intermediate) roots! Especially for those customer facing sites where you can’t control what updates hit the clients and thus potentially might be denied access to your sites.

Recently some colleagues of me logged a case because a GPO which worked fine before didn’t seem to work anymore. In the GPO the security of c: was redefined. In the end the root cause of that problem was McAfee. Whilst troubleshooting with Microsoft, in one of the generated log files we noticed SYSVOL mismatch errors.

When troubleshooting GPO’s often an utility called GPOtool.exe is used. This tool is available in the Windows 2003 resource kit tools. Since that tool I’ve never seen a newer version been released. I always assumed it just worked with Windows 2008 or 2008 R2 Domain Controllers. So when we got the following errors, we assumed we had a problem with some of our GPO’s:

In words: Error: sysvol mismatch. At first sight the version in the output seem identical. Also when verifying using ADSIedit, the GPO Management Console & in the SYSVOL share, all GPO related versions seemed to be correct. One thing we noticed though, only GPO’s which were exported and imported in a newer GPO seemed to be mentioned. Although I see no reason for that to cause versioning issues. After some googling I came across this: http://kb.elmahdy.net/2011/02/gpotool-for-windows-server-2008-r2.html

So it seems Microsoft (internally) has a more recent build of GPOtool.exe which plays nicer with Windows 2008 R2 domain controllers. I am by no means responsible for the tool provided on that blog, but I tested it in my environment and it worked fine. The exe seems to be signed by Microsoft, so I would assume it’s safe. To conclude the correct output:

And some GPOtool.exe version information (gpotool1 is the old one):

P.S. It is to my understanding that the Windows Server 2012 will have a GPMC which has enhanced capabilities regarding the GPO health. Way to go!

In the last three posts:

• Service Accounts: Active Directory Permissions Issues: Part #1 SharePoint
• Service Accounts: Active Directory Permissions Issues: Part #2 Dynamics Ax 2012
• Service Accounts: Active Directory Permissions Issues: Part #3 SQL 2008 R2

I came to the conclusion that given Authenticated Users Read permissions seems to solve some issues. However providing the “full read” permission might be a bit blunt. I was wondering what property is there that the default permissions don’t cover….

I didn’t see the link at first, but suddenly all puzzle pieces fell together. When trying to find a solution for Part 3 of my AD Service Account Permissions Issues I came across these posts which provide an alternative solution:

• Issues with SQL Server in a Windows 2008 Domain
• Windows 2008 and xp_logininfo

They say to add the service account, which requires a higher level of read access on the involved service accounts, to the built-in group “Windows Authorization Access Group”.

When this group hasn’t been moved you can find it in the Builtin container:

And the group we are discussing:

Now how does this help with our case? When adding users to this group they are granted read access to the tokenGroupsGlobalAndUniversal attribute on all users. And this seems to be the exact permission we were looking after! Instead of granting Authenticated Users Read, it would be sufficient to grant them Read to the tokenGroupsGlobalAndUniversal attribute. But then again, that would be a lot of work compared to just adding them to the built-in group.

After some more research the “Pre-Windows 2000 Compatible Access” seems also tightly coupled to this permissions related stuff. My guess is that choices in the past (during the DCpromo) and manual modifications to either of this groups might determine whether or not you are seeing these kind of issues. Here are the members of these groups after a Windows 2008 R2 being DC Promo-ed into a new domain.

• Pre-Windows 2000 Compatible Access: only Authenticated Users are member
• Windows Authorization Access Group: only Enterprise Domain Controllers are member

In my domain neither of these groups had the “Authenticated Users” in them. So that’s why adding the service accounts made sense. It would by my guess that the far easiest workaround would be to add the authenticated users to the pre-windows 2000 compatible access group. After all in a new 2008 R2 domain this is done for you so this would mimic a standard installed domain based on 2008 R2 media. So I would conclude that this isn’t against best practices or that no security holes are being created. Do you agree?

Some more technical background regarding this attribute: KB331951: Some applications and APIs require access to authorization information on account objects

And yep, there’s more instances of this phenomena! I also came across the following when install an Active Directory Federation Services farm which uses SQL to store its configuration. Whilst there was not noticeable impact (yet), I saw the SQL loggings being filled with the following warnings:

In words: The activated proc '[IdentityServerPolicy].[SqlQueryNotificationStoredProcedure-616f6b36-c503-4503-a6cd-7e067a1b9e43]' running on queue 'AdfsConfiguration.IdentityServerPolicy.SqlQueryNotificationService-616f6b36-c503-4503-a6cd-7e067a1b9e43' output the following:  'Could not obtain information about Windows NT group/user '***\s_****_adfs', error code 0x5.'

And a slightly other one:

In words: An exception occurred while enqueueing a message in the target queue. Error: 15404, State: 19. Could not obtain information about Windows NT group/user '***\s_****_adfs', error code 0x5.

Error: 28005, Severity: 16, State: 2.

The solution: is to give the “Authenticated Users”  “Read Permissions” on the ADFS service account. An easy way to test this solution is executing the following query:

The query xp_logininfo ‘Domain\service account’ will return something like this if things go well:

Or like this if the SQL Server service lacks the mentioned permissions:

If you’re interested in a more definite solution which does not involve modifying the security of all your service accounts, make sure to read Service Accounts: Active Directory Permissions Issues: Part #4 Conclusion.

The solution “grant Authenticated Users Read permissions on the involved service accounts” can also be applied during installations of Dynamics Ax 2012. In fact it’s twofold. We came across two instances of this problem. When installing the bits for the Enterprise Portal extensions there’s an option to prepare SharePoint for the deployment of the Ax 2012 Portal. This setup failed with the error “The given key was not present in the dictionary.” This obviously sounds very much like my post in Service Accounts: Active Directory Permissions Issue Part #1. And indeed this change allowed to setup to end gracefully.

But we also came across an other issue which does not seems so related at first sight. When trying to configure the Business Connector Proxy account in the Ax console we received the following error when clicking ok:

In words: Infolog (1). One or more critical STOP errors have occurred. Use the error messages below to guide you or call your administrator. The alias/network domain entered for the Business Connector proxy is not valid.

I don’t know why, but somehow this felt like the same issue as before. And indeed. Granting the “Authenticated Users” “Read” access on the BCP account allowed us to configure it as the BCP user.

If you’re interested in a more definite solution which does not involve modifying the security of all your service accounts, make sure to read Service Accounts: Active Directory Permissions Issues: Part #4 Conclusion.

Currently I’m involved in a project where we are setting up a lot of Windows technologies, just to name a few: Dynamics Ax 2012, BizTalk, SharePoint 2010, FIM 2010 and thus SharePoint Foundation 2010. It seems some legacy thingy in the Active Directory is biting us in the ass, so for both SharePoint (Full blown and Foundation) and Dynamics Ax we’ve had to modify the permissions on the service accounts used by those products.

So here’s the first occurrence I came across. It happened when I was running the initial configuration wizard of SharePoint Foundation 2010. This SharePoint instance will host the FIM 2010 R2 Portal in the near future. This wizard is to be executed after installing the bits & bytes of SharePoint 2010.

What it says in words: Failed to create the configuration database. An exception of type System.Collections.Generic.KeyNotFoundException was thrown. Additional exception information: The given key was not present in the dictionary.

One thing to look into would be the permissions on the SQL server instance. All in vain. In the end I fired up google and came across numerous posts like these:

• “The given key was not present in the dictionary” issue in Sharepoint 2010 RTM
• The given key was not present in the dictionary Error When Validating User Accounts

The solution is pretty simple to implement, but I’m still struggling whether there’s a “nicer” way which does not involve touchy every service account or the OU in which they reside. Using Active Directory and Computers, or the new Active Directory Administrative Center, we can modify the permissions on the involved service accounts. The ones involved or the ones which are being used by SharePoint: to run various service, application pools or the farm admin. In order to view the security tab on a given user you might have to enable the advanced features in the view option of the ADUC MMC.

Once you got your service account open, just check “Allow Read” for “Authenticated users”.

This exact same error also popped up when registering a new Managed Service Account within the Central Administration site:

If you’re interested in a more definite solution which does not involve modifying the security of all your service accounts, make sure to read Service Accounts: Active Directory Permissions Issues: Part #4 Conclusion.

A customer of mine wanted to use the built-in windows update of Windows Server 2008 R2 to apply the latest security patches. When they clicked check for updates they received the following error:

In words: An error occurred while checking for new updates for your computer. Error Code 80072EE2

I’ve came across this in the past and I know google would lead me to a solution in 1-2-3: http://tmgblog.richardhicks.com/2010/08/07/running-windows-update-on-a-tmg-firewall-fails-with-result-code-80072ee2/ The solution mentioned over there is to point the SYSTEM to the TMG for http requests. This can be done by executing the following command from an elevated command prompt:

• netsh winhttp set proxy localhost:8080 [when 8080 is your proxy port]

Thinking all would by fine I clicked try again.  And guess what, it failed again, but now with a slightly different error: 80072F8F. I googled and googled, tried some workaround for which I doubted they would work, but finally I came across this page: http://bent-blog.de/wsus-fehler-bei-der-synchronisierung-fehler-0x80072f8f-bei-windows-update/ I only read it diagonally as I can manage to understand German, but not that great. But I saw some screenshots which resembled a lot to my case.

The reason seems to be the absence of a root certificate in the trusted root certificate store. You can easily verify this by opening https://www.update.microsoft.com on the TMG server. If you receive a certificate error, you might be suffering the same problem. I just took the missing root certificate (GTE CyberTrust Global Root) and added it to the trusted root certificates of the computer certificate store. And voila!

I had a colleague which was configuring Symantec Mobile Device Management. The event log was showing messages of a service trying to connect to an URL but failing with 407 proxy authentication required. As far as we knew no proxy settings were configured. Well they were at a point, but not now…

Here’s what led me to the solution: http://www.symantec.com/business/support/index?page=content&id=TECH127695

Delete these keys:

• HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\DefaultConnectionSettings
• HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections\SavedLegacySettings

[Update] that same day I came across an identical issue but in SCCM with the Active Update Point configuration push towards WSUS.

From the WCM.log:

This SRVSCCM02.rdsolutions.local system is the Top Site where WSUS Server is configured to Sync from Microsoft Update (WU/MU) OR do not Sync.~ 
$$<SMS_WSUS_CONFIGURATION_MANAGER><02-23-2012 15:33:29.066-60><thread=4020 (0xFB4)>
Found WSUS Admin dll of assembly version Microsoft.UpdateServices.Administration, Version=3.0.6000.273, Major Version = 0x30000, Minor Version = 0x17700111~ 
$$<SMS_WSUS_CONFIGURATION_MANAGER><02-23-2012 15:33:29.120-60><thread=4020 (0xFB4)>
Found WSUS Admin dll of assembly version Microsoft.UpdateServices.Administration, Version=3.1.6001.1, Major Version = 0x30001, Minor Version = 0x17710001~ 
$$<SMS_WSUS_CONFIGURATION_MANAGER><02-23-2012 15:33:29.121-60><thread=4020 (0xFB4)>
The installed WSUS build has the valid and supported WSUS Administration DLL assembly version (3.1.7600.226)~ 
$$<SMS_WSUS_CONFIGURATION_MANAGER><02-23-2012 15:33:29.121-60><thread=4020 (0xFB4)>
System.Net.WebException: The request failed with HTTP status 407: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. ).~~   at Microsoft.UpdateServices.Administration.AdminProxy.CreateUpdateServer(Object[] args)~~   at Microsoft.UpdateServices.Administration.AdminProxy.GetUpdateServer(String serverName, Boolean useSecureConnection, Int32 portNumber)~~   at Microsoft.SystemsManagementServer.WSUS.WSUSServer.ConnectToWSUSServer(String ServerName, Boolean UseSSL, Int32 PortNumber)  $$<SMS_WSUS_CONFIGURATION_MANAGER><02-23-2012 15:33:34.210-60><thread=4020 (0xFB4)>
Remote configuration failed on WSUS Server.~ 
$$<SMS_WSUS_CONFIGURATION_MANAGER><02-23-2012 15:33:34.226-60><thread=4020 (0xFB4)>

In the component status overview log:

And in words:

WSUS Synchronization failed.
Message: WSUS server not configured. Please refer to WCM.log for configuration error details..
Source: CWSyncMgr::DoSync.
  The operating system reported error 2147500037: Unspecified error

Introduction

Lately a colleague of mine was struggling with the following: he wanted a script to be ran from within a Startup Script GPO. Now the problem he was encountering was the following Security Warning from PowerShell:

In words:

Security Warning
Run only scripts that you trust. While scripts from the Internet can be useful, this script can potentially harm your
computer. Do you want to run \\file.setspn.com\scripts\script.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"):

Now if you have to run something like this occasionally, you can type R and get on it with it. However if this is a script which has to automatically run at startup, this will be a showstopper. So here is some additional information on how to avoid these warnings.

I didn’t tested this in the context of a computer starting up and executing such a script. I just tested this from the Administrator point of view: you run a PowerShell script from a share and you want to avoid that warning. Now what? I will briefly explain Execution Policies and Execution Policy Scopes before actually presenting a solution.

PowerShell Execution Policies

As most of us know by now, PowerShell comes with an execution policy. This policy define which scripts can ran and from which location. By default it’s configured to restricted:

In words:

File \\file.setspn.com\scripts\script.ps1 cannot be loaded because the execution of scripts is disabled on this system.
Please see "get-help about_signing" for more details.
At line:1 char:37
+ \\file.setspn.com\scripts\script.ps1 <<<<
    + CategoryInfo          : NotSpecified: (:) [], PSSecurityException
    + FullyQualifiedErrorId : RuntimeException

However we can easily change the policy to one of the following options:

• Restricted: no scripts
• AllSigned: only signed scripts
• RemoteSigned: local [+detected as local intranet] scripts and signed scripts remotely
• Unrestricted: all scripts but comes with warnings when running from a share
• Bypass: all scripts, no warnings

If you want a full explanation on each of these policies, check TechNet: about_Execution_Policies

PowerShell Execution Policy Scope

Now if you changed the above policy using the Set-ExecutionPolicy Policy command, you changed it for everyone running PowerShell scripts on the current machine. Did you know there is an alternative? You can actually specify a scope using the –scope parameter! This can come in very handy when you just want to open up PowerShell temporary for the installation of a product. The following scopes exist:

• Process: affects only the current PowerShell session
• CurrentUser: affects only the current user
• LocalMachine: affects all users on the current computer

If you want a full explanation on each of these scopes, check TechNet: about_Execution_Policies There’s also additional info available if you want to control this by using GPO.

Determing The Active PowerShell Execution Policy

Using the Get-ExecutionPolicy command you can get the current policy, however if you add the –list parameter you can see where it’s coming from:

Now after running Set-ExecutionPolicy –Scope Process Unrestricted

Now the advantage is that whenever this PowerShell session is closed, the overall policy remains to what is was before my modification.

Getting Rid Of the Warning

All of the options below will explain how you can get rid of the warning when execution a script from an UNC path. Between [] I’ve added the PowerShell execution policies this is working with. The PowerShell script I’m running is located on \\file.setspn.com\scripts and it only contains one line: “write-host –forefgroundcolor green “script executed successfully””.

Option 1: use a shortname in the path [RemoteSigned/Unrestricted]

Why does this work?

By default IE7 and up are configured to automatically detect intranet network. This one checkbox actually is equivalent to checking the 3 options below it. Because we use a UNC path with a short name (\\file), it’s detected as an Intranet and hence no warning is displayed. However once you use a FQDN (\\file.setspn.com), this detection does not work. Reference: Enabling "Automatically Detect Intranet Network" on a domain member computer will enable all the three Intranet Options automatically.

Do we like this? I don’t. I hate NetBIOS names. On to the next option

Option 2: Specify Local Intranet Sites [RemoteSigned/Unrestricted]

The security warning only comes up when the script is ran from an untrusted location, we can add the UNC path to the local intranet. We can do this using one of the following formats:

1. \\*.setspn.com
2. file://*.setspn.com
3. \\file.setspn.com
4. file://file.setspn.com
5. *.setspn.com
6. file.setspn.com

The first 4 options will only affect files accessed from UNC paths, however option 5 & 6 will also involve HTTP/HTTPs traffic. Option 1 & 2 are in fact the same. The same goes for option 3 & 4. Whenever you enter an UNC path like \\location it will automatically be converted to file://location by IE. After adding \\*.setspn.com:

The script now runs just fine:

Option 3: Copy The Script Locally [RemoteSigned/Resctricted]

Obviously this is not applicable in certain scenario’s, but as a quick work around it’s a possibility.

Option 4: Sign your scripts [AllSigned/RemoteSigned/Resctricted]

Another possibility is to sign your scripts. A detailed guide is out of the scope of this post, a good description of the process can be found here: Scott Hanselman: Signing PowerShell Scripts

Option 5: Use The Bypass ExecutionPolicy [Resctricted/AllSigned/RemoteSigned/Resctricted]

All of the above options (beside #1) require you to do some modifications. There’s also a way to just suppress these warnings: the Bypass exeuction policy! I think you can apply it in one of the following ways:

1. Set-ExecutionPolicy Bypass
2. Set-ExecutionPolicy Bypass –Scope Process
3. Powershell.exe –ExecutionPolicy Bypass –file \\file.setspn.com\script.ps1

Option 1 just doesn’t feel right. I myself would definitely go for option 2 or 3. They are more or less equivalent. Option 3 is ideally for calling a PowerShell script from within a bat file. Option 2 is just neat as it’s only active temporarily.

Conclusion

I’m not favoring one of the above options. I think PowerShell execution policies should be part of the Workstation/Server design. A given policy should be decided upon and pushed through GPO. Once that baseline is established, I would choose one of the above options to make a certain script work in the given scenario.

Like if RemoteSigned would be active on workstations, I would consider adding the UNC path to the Local Intranet sites. On the other hand if I would be administering servers (RemoteSigned active) and I’d have a script which I have to run just once, I would consider changing the execution policy to Bypass just for this PowerShell.exe instance (-scope process).

Happy PowerShelling!

One of the things a colleague of mine encountered in the past, and which I stumbled upon lately is the following. Sometimes people want to have the Local Administrator account disabled on their servers. There has been a GPO to do this for ages. It’s located below Computer Settings > Windows Settings > Security Settings > Local Policies > Security Options. The setting is “Accounts: Administrator Account Status”: Disabled.

The screenshot shown below is from the security policy on a server which has the policy (Administrator Status: disabled) applied. You can see that A group policy is setting the setting to enabled. Which is in fact the opposite of what I have configured through the GPO.

One could think I have another GPO being applied later. But using gpresult /H:report.html I can clearly see “my” GPO is winning and that the setting in fact should be set to disabled…

Also a regular Resultant Set Of Policy shows the setting as disabled…

But the account is Active and remains in this state…

So, Group Policy Preferences to the rescue! It’s not a real answer as to why things are going wrong, but it’s definitely a doable workaround. This policy works flawless.

You can’t always get to the bottom of things…

1. Use GPMC GPO Backup Feature To Locate Unresolvable SIDs

Sometimes you might have GPO’s which reference SID’s which cannot be resolved. Their might be various reasons for that. Someone might have configured the GPO to reference a certain account in a setting whilst that account was deleted somewhere in time afterwards. Or like I encountered: you use GPO backups to import & export your GPO’s from a lab to an acceptance environment and you simply forget to translate some of the SIDs.

A neat trick which I found out by accident is the “Backup All…” GPO option from the Group Policy Management Console. This will try to resolve all accounts used in your GPO’s and throw a warning if there’s a problem. You could do this every now and then to keep your GPO’s squeaky clean.

2. Generate an HTML Report Of All Your GPO’s

Whenever you’re documenting your GPO’s, or you simply want to have a snapshot in time of the settings, versions, links, security, …. you can choose to create a GPO report from the GPMC. Using PowerShell however you can issue the following command to get a single-file HTML which will nicely give you all the required information. It would perhaps be a nice idea to run this monthly or even more frequently if you want to have some auditing trail as to what is changed. But if you really need this, I think AGPM will be a better fit.

Get-GPOReport -All -Domain contoso.com –Server dc01 -ReportType HTML -Path C:\Users\thomas.vuylsteke\Desktop\GPO_Report\GPO_Report.html

The following screenshot shows an example of the layout. By default everything, except the subsections of each GPO, is hidden. You can easily scroll from GPO to GPO, and I can imagine it’s very simple to edit the HTML file if you only want a subset of the policies in your report.

This is going to be a short one: believe it or not, but there’s an alternative to installing Wireshark (or at least winpcap) on all your servers when troubleshooting network related issues. It actually has been out there ever since Windows 7 and Windows 2008 R2 came out. Using “netsh trace start capture=yes” you can start capturing network traffic. Simply execute “netsh trace stop” whenever you are ready. The result is an .etl file you can open using network monitor 3.2 or later. Awesome #1: no need to install anything!

But it doesn’t stops there: using “netsh trace start capture=yes persistent=yes” tracing will even continue after a reboot. So as soon as the network interface starts communicating, traffic will be captured! Awesome #2: capturing information during the boot process of a server!

Some screenshots:

Some additional information to get you started: http://blogs.technet.com/b/netmon/archive/2009/05/13/event-tracing-for-windows-and-network-monitor.aspx

But it doesn’t stops here! IE 9, currently in beta, has a new addition too: a built-in network trace utility! In fact it provides (some?) of the functionality Fiddler provides. I am not that experienced with Fiddler, so I don’t know how to compare them feature-wise, but It’s definitely useful having this built-in. Checking it out for sure!

To start using it: Press F12 –> Network –> Start Capturing

P.S. I am by no means saying you should ditch Wireshark/ Fiddler for Netsh Trace/ IE 9 network tracing.  But I think they are great tools to have in your toolbox!

Happy Tracing, Happy New Year!
Thomas

In the project I’m currently involved in, we have several virtual machines installed with Windows 7 as the operating system. These workstations are used as administration workstations to manage the server infrastructure. Every once and a while someone would get disconnected from their terminal server session. By simply reopening the session they could continue their work. We stumbled upon the following KB article: KB2083411 The article states:

When the policy is refreshed (by default, every 90 minutes, or manually through GPUPDATE), the policy settings are deleted and then reset. During this period, the configuration on the server is temporarily valid. Therefore, all sessions may be disconnected

Because we were enabling remote desktop through GPO, this was the exact issue we were having.  We could reproduce it by executing gpupdate in a remote desktop session. The disconnection would not occur every time, but every once in a while. To be more precise when enabling remote desktop through group policies (Allow users to connect remotely using Terminal Services ), the following registry key is set:

The workaround suggested in the KB article is to set the registry key fDenyTSConnections below HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server to a value of 0. Basically this is just enabling remote desktop through the registry instead of through the GUI.

If you have to do this on a lot of workstations there are a few options. We first thought to just deploy this using our desktop deployment solution (SCCM). However a colleague of mine had the great idea to just try setting it using group policy preferences. Whenever group policies are being refreshed, everything below HKLM\SOFTWARE\Policies is erased and reset, but group policy preferences operate in an other way. Whenever you set something with group policy preferences, it’s set forever, unless you check the option “remove this item when it is no longer applied”. For some additional information on the differences between GP policy setting and GP preference settings.

Here are also some related topics on the TechNet forums: http://social.technet.microsoft.com/Forums/en/winserverGP/thread/10fb967c-c6c8-480b-8d30-70f0da15cdba and http://social.technet.microsoft.com/Forums/en/winserverGP/thread/cd94ea99-a843-4781-bbcf-7538182511c9