Earlier this week I’ve been talking to a customer about the “Protected Users” group. You might have seen it appearing when introducing the first 2012 R2 domain controller. Here’s a good explanation on its purpose:

Protected Users is a new global security group to which you can add new or existing users. Windows 8.1 devices and Windows Server 2012 R2 hosts have special behavior with members of this group to provide better protection against credential theft. For a member of the group, a Windows 8.1 device or a Windows Server 2012 R2 host does not cache credentials that are not supported for Protected Users. Members of this group have no additional protection if they are logged on to a device that runs a version of Windows earlier than Windows 8.1. Source: TechNet: How to Configure Protected Accounts

The above is actually a bit misleading. The functionality was actually backported to Windows 2008 R2/Windows 2012 in the hotfix KB2871997 See blogs.technet.com: An Overview of KB2871997 for an explanation on this.

This group might be part of your organization’s strategy to reduce the attack surface for pass the hash. A great white paper on this can be found here: Mitigating Pass-the-Hash (PtH) Attacks and Other Credential Theft, Version 1 and 2

One of the things the Protected Users group ensures is that no NTLM hashes are available to be used or stolen. Now I wanted to see this for myself. There are various tools out there that are capable of listing the various secrets. I tried Windows Credential Editor (WCE) but that one didn’t work on (my) Windows 2012 R2. So I used Mimikatz. My setup: A 2012 R2 domain controller and a 2012 R2 member server. I’ve got 3 domain admins: one that has the remote desktop session open to the member server and then two that have a powershell runnning through runas. Of the latter one is a member of the Protected Users group:

Run as different user: SETSPN\john

Run as different user: SETSPN\thomas

As you can see John is an oldschool Domain Admin whereas Thomas has read the Mitigating PtH whitepaper and is a proud member of the Protected Users group. This is the PowerShell oneliner I used to dump the groups I care about: WHOAMI /GROUPS /FO CSV | ConvertFrom-Csv | where {$_."group name" -like "Setspn\*"}

Here you can see the Protected Users admin has no NTLM available:

Where the regular admin has NTLM available:

Here’s the difference from an attacker point of view:

Start Mimikatz –> Privilege::debug –> sekurlsa::logonpasswords And here are the goodies:

John:

Authentication Id : 0 ; 3529276 (00000000:0035da3c)
Session           : Interactive from 0
User Name         : john
Domain            : SETSPN
Logon Server      : SRVDC01
Logon Time        : 2/24/2016 6:59:54 PM
SID               : S-1-5-21-4274776166-1111691548-620639307-5603
        msv :
         [00000003] Primary
         * Username : john
         * Domain   : SETSPN
         * NTLM     : 59884edfb057d0fec8cb7e0d571dc200
         * SHA1     : 7e655db2b3a7e88fb0c50ca56416ae655469f09e
         [00010000] CredentialKeys
         * NTLM     : 59884edfb057d0fec8cb7e0d571dc200
         * SHA1     : 7e655db2b3a7e88fb0c50ca56416ae655469f09e
        tspkg :
        wdigest :
         * Username : john
         * Domain   : SETSPN
         * Password : (null)
        kerberos :
         * Username : john
         * Domain   : SETSPN.LOCAL
         * Password : (null)
        ssp :
        credman :

Thomas:

Authentication Id : 0 ; 3493146 (00000000:00354d1a)
Session           : Interactive from 0
User Name         : thomas
Domain            : SETSPN
Logon Server      : SRVDC01
Logon Time        : 2/24/2016 6:59:36 PM
SID               : S-1-5-21-4274776166-1111691548-620639307-5602
        msv :
         [00010000] CredentialKeys
         * RootKey  : db1c2347608db0c4e2d89bbd6c328bf6f42671b7d88653cd4cc9af2713
e958f0
         * DPAPI    : 63adfe49948fca81c885933b3aa23eba
        tspkg :
        wdigest :
         * Username : thomas
         * Domain   : SETSPN
         * Password : (null)
        kerberos :
         * Username : thomas
         * Domain   : SETSPN.LOCAL
         * Password : (null)
        ssp :
        credman :

As you can see the admin that’s a member of the Protected Users group does NOT have the NTLM hashes dumped. Wooptiedoo! Now think and test before you start adding the Domain Admins group to the Protected Users group! By no means you should do that! Here’s some good information on how to start with the Protected Users group and some additional caveats: How to Configure Protected Accounts

Here’s one from my side: after adding my admin user to the Protected Users group he was no longer to RDP to a 2012 R2 member server:

In words: A user account restriction (for example, a time-of-day restriction) is preventing you from logging on. For assistance, contact your system administrator or technical support.

Remote desktop to a Windows 2008 R2 worked fine with that account. It seems for my Protected User admin to be able to log on to a Windows 2012 R2 server it had to actualy use mstsc.exe /restrictedadmin and I had to enable Restricted Admin mode on the member server:

You can find that value below HKLM\SYSTEM\CurrentControlSet\Control\Lsa

If you want to know more about the Protected Users group and the Restricted Admin feature read up on both of them here: TechNet: Credentials Protection and Management or digital-forensics.sans.org:Protecting Privileged Domain Accounts: Restricted Admin and Protected Users

Some additional reading on Restricted Admin mode: Restricted Admin mode for RDP in Windows 8.1 / 2012 R2

I’ve got a customer running some virtual machines on the Azure IAAS platform, and when doing a quick checkup I found the following recurring events in the event log:

• The volume (C:) was not optimized because an error was encountered: Neither Slab Consolidation nor Slab Analysis will run if slabs are less than 8 MB. (0x8900002D)
• The volume Temporary Storage (D:) was not optimized because an error was encountered: Neither Slab Consolidation nor Slab Analysis will run if slabs are less than 8 MB. (0x8900002D)
• The volume Data (F:) was not optimized because an error was encountered: Neither Slab Consolidation nor Slab Analysis will run if slabs are less than 8 MB. (0x8900002D)

Sample event:

I’m aware that Windows does maintenance on a regular base all by itself. One of these tasks is a scheduled defrag:

You’ll probably see that there’s no trigger for this particular task, yet it runs regularly! That’s because there’s another scheduled task called “regular maintanance” who calls this specific task. An excellent writeup can be found here: DataToHelpThePeople: Maintenance at 3AM in the Morning

I did some googling and I quickly came accross this: KB2964429: Storage Optimizer memory use increases when it runs on thin provisioned LUNs

From that article: There's no need to run Storage Optimizer on thin provisioned LUNs that use an allocation size (also known as slab size) of less than 8 MB. Thin provisioned LUNs that have a smaller slab size manage space more efficiently, and the benefits of defragmenting them are not as great.

So that got me curious, how big is the allocation size on these volumes? We can find this information using diskpart:

After selecting a given volume, execute “filesystem”. In our example we have 4 MB which is, as the erorr states, less than 8 MB. Both the OS disk and the Temporary Disk have 4 MB which is the default I guess. My custom disk (F:) also has this value.

If we check the action of the scheduled task we can see that this is the command being executed: %windir%\system32\defrag.exe -c -h -k -g –$ If we execute that command manually, the same events are logged.

From defrag.exe /? I can tell that we could ommit –k in order to avoid slab analysis and consolidation. However upon executing the command I’m not really sure much happens in the background.

In the following screenshot you can clearly see that for each volume an event with ID 258 is logged: The storage optimizer succesfully completed retrim on XYZ. This events are NOT logged when I run the defrag command without the –k switch.

Your first reaction might be to disable this scheduled task all togehter. You might not care about fragmentation. But you might care about your billing statement… If you care check: fabriccontroller: Release unused space from your Windows Azure Virtual Hard Disks to reduce their billable size Bottom line: whenever this maintenance tasks finds unused storage it reclaims it and lowers the overall VHD size. Which means you pay less for your storage account!

I’m not really sure how to move forward. On the one hand I really detest recurring, safe to ignore, errors in events logs. But I don’t like changing system stuff like this without some official guidance. So if you got something to add, feel free to comment!

One of my customers had problems with certain users being member of a lot of Active Directory groups. This resulted in several client side issues. There’s an easy and well-known “fix” for that: raise the MaxTokenSize registry key on all Windows operating systems in your domain. On Windows 8(.1) / 2012 (R2) the MaxTokenSize is already at its maximum (advised) value out of the box. That value is 48.000 bytes. In order to mitigate these users their access problems we raised the MaxTokenSize to 48.000 bytes on all clients and servers that are running Windows 7/ Windows 2008 R2. After this change the typical issues were gone. However new ones came up:

From time to time, when HTTP is involved, issues were encountered:

• Opening the Direct Access management console (depends on WinRM)
• Open the FIM Portal
• Streaming App-V packages over HTTP
• …

Typically the user would receive several authentication prompts and even after specifying valid credentials another prompt would reappear. Example browser based issue:

As you can see the browser gives an HTTP 400 Bad Request error. Using a network trace we can easily see why it’s considered bad:

And the packet details:

The details clearly state that The size of the request headers is too long.

The problem here is that the token is allowed to be up to 48.000 bytes where it used to be 12.000 bytes. The http subsystem of a windows server has several parameters that are supposed to protect the server from oversized requests. However, as the token can now be a lot larger, the maximum request size has to be tuned as well:

From: KB820129

Below: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters there are two interesting values:

And from: KB2020943 we can find a formula to calculate the MaxFieldLength to set based on the MaxTokenSIze.

If MaxToken is 48.000 bytes (default in Windows 2012 and configure by GPO for 2008 R2/ Win7):

• (4/3 * 48000) + 200 = 64200

We’ll use the maximum allowed value of MaxFieldLength 65534 (=~ 64200) to allow tokens up to 48000 bytes. We’ll also use this value for MaxRequestBytes.

• MaxFieldLength: we can take the maximum allowed value: 65534
• MaxRequestBytes:  65534

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters

Other useful information:

• http://blogs.msdn.com/b/ashishsingh/archive/2010/04/03/windows-authentication-for-accounts-with-large-kerberos-tickets-may-not-work-despite-having-maxtokensize-in-place.aspx
• http://www.grouppolicy.biz/2013/06/how-to-configure-iis-to-support-large-ad-token-with-group-policy/

I specifically wanted to post this information as in many other only articles/posts I always see people just using the maximum allowed value for MaxRequestBytes and I don’t feel 100% comfortable with that. Second,  in my opinion it’s advised to have these values pushed out to all your server systems. Especially now that Windows 2012 and up have a MaxTokenSize of 48.000 by default. If you don’t push these HTTP.sys parameters, you’ll end up troubleshooting the same phenomena multiple times from different angles. Why waste time?

This morning I wanted to have a quick look at our Direct Access infrastructure and when opening the console I got greeted with various errors all explaining that there was a configuration load error:

In words: ICMP settings for entry point cannot be determined. Or:

In words: Settings for entry point Load Balanced Cluster cannot be retrieved. The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.

Because initially I only stumbled upon the ICMP settings … error I had to dig a bit deeper. I searched online on how to enable additional tracing capabilities, but I couldn’t find anything. From reverse engineering the RaMmgmtUI.exe I could see that more than enough tracing was available. Here’s how to enable it:

Create a REG_DWORD called DebugFlag below HKLM\SYSTEM\CurrentControlSet\Services\RaMgmtSvc\Parameters. For our purpose we’ll give it a value of 8. For an overview of the possible values:

I’m not sure if you can combine those in some way. After finding this registry key, I was able to find the official article on how do this: TechNet: Troubleshooting DirectAccess. I Should have looked a bit better for that information perhaps… After closing the Remote Access Management Console and opening it again, the log file was being filled up:

You can find the trace file in c:\Windows\Tracing and it’s called RaMgmtUIMon.txt After opening the file I stumbled across the following error:

2112, 1: 2014-09-30 11:51:43.116 Instrumentation: [RaGlobalConfiguration.AsyncRefresh()] Exit
2112, 12: 2014-09-30 11:51:43.241 ERROR: The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.
2112, 9: 2014-09-30 11:51:43.242 Failed to run Get-CimInstance

I then used PowerShell to try to do the same: connect to the other DA node using WinRM:

The command: winrm get winrm/config –r:HOSTNAME The error:

WSManFault
    Message = The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.

Error number:  -2144108297 0x803380F7
The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid.

Googling on the error number 2144108297 quickly got me to the following articles:

• MIRU.CH: How the kerberos token size can affect winrm and other kerberos based services
• KB820129: Http.sys registry settings for Windows
• KB970875: Large Kerberos tokens cause WinRM requests to fail

Basically I was running into this issue because my AD user account was member of a large amount of groups. The MaxTokenSize has been raised in Windows 2012 (R2) so that’s already covered, but winhttp.sys, which WinRM depends on, hasn’t. When running into Kerberos token bloat issues on web applications, typically the MaxRequestBytes and MaxFieldLength values have to be tweaked a bit.

There are various ways to configure these. Using GPO or a manual .reg file that you can just double click:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters]

"MaxRequestBytes"=dword:0000a000

"MaxFieldLength"=dword:0000ff37

In my environment I’ve set MaxRequestBytes to 40960 and MaxFieldLength to 65335. But I am by no means saying those are the advised values. It’s advised to start with a lower value and slightly increase until you’re good to go.

Conclusion: if you run into any of the above errors when using the Direct Access management console, make sure to check whether WinRM is happy. In my case WinRM was in trouble due to the size of my token.

Recently I helped a customer which was having troubles migrating from a Windows 2003 cluster to a Windows 2012 cluster. The resources they were running on the cluster consisted of many in house developed applications. There were about 80 of them and they were running as generic applications.

Due to Windows 2003 being end of life they started a phased migration towards Windows 2012 (in a test environment). At first the migration seemed to go smooth, but at a given moment they were only able to start a limited amount of applications. The applications that failed gave an Out Of Memory exception (OutOfMemoryException). Typically they could start about 25 applications, and from then on they weren’t able to start more. This number wasn’t exact, sometimes it was more, sometimes it was less.

As I suspected that this wasn’t really a failover clustering problem but more a Windows problem I googled for “windows 2012 running many applications out of memory exception”. I found several links:

HP: Unable to Create More than 140 Generic Application Cluster Resources

IBM: Configuring the Windows registry: Increasing the noninteractive desktop heap size

If the parallel engine is installed on a computer that runs Microsoft Windows Server, Standard or Enterprise edition, increase the noninteractive desktop heap size to ensure that a sufficient number of processes can be created and run concurrently

So it seems you can tweak the desktop heap size in the registry. Here is some background regarding the modification we did to the registry.

The key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

SharedSection has three values: (KB184802: User32.dll or Kernel32.dll fails to initialize)

• The first SharedSection value (1024) is the shared heap size common to all desktops. This includes the global handle table, which holds handles to windows, menus, icons, cursors, and so forth, and shared system settings. It is unlikely that you would ever need to change this value.
• The second SharedSection value (3072) is the size of the desktop heap for each desktop that is associated with the "interactive" window station WinSta0. User objects like hooks, menus, strings, and windows consume memory in this desktop heap. It is unlikely that you would ever need to change this second SharedSection value.
• The third SharedSection value (512) is the size of the desktop heap for each desktop that is associated with a "noninteractive" window station. If this value is not present, the size of the desktop heap for noninteractive window stations will be same as the size specified for interactive window stations (the second SharedSection value).

Default on Windows 2012 seems to be 768

Raising it to 2048 seems to be a workaround/solution. A reboot is required! After this we were able to start up to 200 generic applications (we didn’t test more). However after a while there were some failures, but at first sight quite limited. This might be due to the actual memory being exhausted. Either way, we definitely saw a huge improvement.

Disclaimer: ASKPERF: Sessions, desktops and windows stations

Please do not modify these values on a whim. Changing the second or third value too high can put you in a no-boot situation due to the kernel not being able to allocate memory properly to even get Session 0 set up

Bonus info: why didn’t the customer didn’t have any issues running the same workload on Windows 2003? They configured the generic applications with “allow desktop interaction”. Something which was removed from the generic applications in Windows 2008. Because they had “allow desktop interaction” configured, generic applications were running in an interaction session and thus were not limited by the much smaller non-interactive desktop heap size.

Recently I worked together with some Microsoft consultants to implement a Direct Access proof of concept. Initially the configuration went just fine. However, the operational state page showed that we had an issue with the IPSEC certificate. Initially we thought the problem had to do with our DA server being unable to reach the internet in order to download the CRL (Certificate Revocation List). So we had two options: either ask the network team to open up the DA’s external interface connectivity towards the internet or configure the DA server so it could use a proxy.

Due to the workload of the network team, not the best reasoning though, we choose option two: the proxy. We configured the proxy in IE and we were able to successfully download the CRL file. The error didn’t go away though. So we figured we had to configure the system to use the proxy. This can be done using netsh:

• netsh winhttp set proxy 10.0.10.30:8080

The error didn’t went away and it was time for lunch so we thought we’d give the DA server a reboot and some time. When we came back there was something weird with the operation state page. It just said Name_Of_DA_Server is not working properly. We tried rerunning the configuration wizard but we got greeted with the following error:

In words: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. Suddenly we figured out that our netsh winhttp proxy might have broken things a bit. So we gave the command a little twist:

• netsh winhttp set proxy 10.0.10.30:8080 bypass-list="<local>;*.contoso.local"

Now that we used this command our DA console was back to normal, but our IPSEC certificate was still in error. The solution was actually quite simple: just make sure the DA server also has a certificate based upon the Computer (or your custom computer) template. Such a certificate can be used for both client and server authentication. This certificate should be issued by the same CA that issues certificates for your clients and that is configured in the DA configuration wizard. After enrolling a certificate we now had a working DA setup. We could see our clients setting up a connection successfully.

One thing that bothered us though: on the operation state page we had an error on the availability of the IPHTTPS interface of the DA server. Now that was odd. netsh showed it as available and after all we had several clients connected to it. When we removed the proxy (netsh winhttp reset proxy) and then the status page went all green!

So for now I’m not even sure if the DA server needs to check the CRL’s of the certificate used by IPHTTPS. But I guess we’ll find out soon if DA stops working. All I can say: the netsh winhttp set proxy doesn’t seems a really good idea to use. Things started breaking in weird and unpredictable ways.

I’m not sure whether there will be another person ever running into this, but if you do I hope this saves you some time.

One of my customers is deploying Windows 8.1 clients. They mainly use SCCM and App-V as a software delivery solution. Besides that they also have some applications published over RemoteApp. Windows 8 has a GPO setting which allows you to configure the RemoteApp Connection URL: Setting the default RemoteApp connection URL on your clients using GPO

My customer mentioned to me that some of the clients got the following error message: There are currently no connections available on this computer.

Verifying the RemoteApp and Desktop Connections event log showed no entries. I double checked that the GPO was applying by checking the following registry key:

• Key: HKEY_CURRENT_USER\Software\Policies\Microsoft\Workspaces
• Value: DefaultConnectionURL: https://termsrv.contoso.local/rdweb/Feed/webfeed.aspx

I did some googling and stumbled upon the following articles:

• RemoteApp Default Connection URL Group Policy not working
• Default Connection URL via GPO 2012 server / windows 8 client

The first one didn’t apply for our situation. Our clients were not part of any VDI or TS setup. The second one seemed interesting. Sadly no solution was given, but it gave us the hint that adding the user to the local administrators might help. And indeed this made the RemoteApp Connection URL work. If adding a user to the local administrator is a solution for any given problem, this means you got to tweak one of the below:

• Permissions in the registry
• Permissions on the file subsystem
• User Rights Assignment (GPO/Secpol.msc)

I removed my user from the Local Administrators and started testing again. Using process monitor I couldn’t see any particular access problems for either file or registry. What I did learn from process monitor though is that a part of the RemoteApp configuration is handled by a group of scheduled tasks! I could clearly see these being created as an Administrator, but not for my regular account. Here’s how they should look:

So somehow the creation of the scheduled tasks went sideways. I thought I ‘d be smart and I started with enabling all failure audits for all events in the security event log: auditpol /set /category:* /failure:enable. Whilst I could see some events being generated during my logon, I couldn’t relate any of those to the scheduled tasks not being created. So no clues there. The only thing I figured I could do is start examining the user right assignment settings. This customer has implemented the security recommendations (Security Compliancy Manager) which means a lot of these are customized. After I put my client in an OU with NO GPO’s whatsoever I could see that my user got this RemoteApp configured just fine.

I thought this would be an easy job. I started comparing my user right assignments on the client (with no GPO’s) and the ones being set by GPO. Typically I would need to hunt something down where by default users have the right, but where the GPO is more strict and only Administrators have it. After going through the list I had found none of those…

So I cloned the GPO, made sure the original one no longer applied to my client and started setting settings to not configured in the user rights assignment section. Sadly I started at the bottom because the one which seemed to be the culprit is Act as part of the operating system. The GPO seemed to set grant this right to “Authenticated Users”. That made me frown… It seems like a very privileged thingy to grant to authenticated users… From the SCM 3.0 toolkit you can see that both the default and the advised setting is “None”:

It seems that somehow the GPO got wrongly configured at some point in time. By default this is set to “None”. After removing the authenticated users from this particular setting I was finally able to get my user his RemoteApp configuration up and running:

This is the setting which seemed to be having this negative effect:

I ‘m still trying to wrap my head around the fact that whilst this setting had authenticated users in it, only non-administrator users were impacted. Either way, case closed!

For those interested in the look and feel of the new Web Application Proxy role, here’s some screenshot of a fairly simple next next finish setup.

The installation:

Remark: seems I’ll have to add a server to my lab environment

The Configuration:

The Management Console:

Open the Remote Access Management Console

The Publish New Application Wizzard:

Remark: read the explanation of the ADFS selection bullet, it’s fairly descriptive.

Seems like basic internal <> external stuff.

Remarks:

• Active Directory Federation Services and Web Application Proxy can’t be combined on one server.
• Active Directory Federation Services is to be installed in your domain before you can install the Web Application Proxy as you need to specify it.
• Selecting Pass-Through on the Preauthentication screen will skip the Relying Party selection and then your application will handle the authentication. This will break your users their SSO experience though.