3 comments

Quick Tip: Enumerate a User his AD Group Memberships

Published on Thursday, August 28, 2014 in ,

Using the two following commands you can easily retrieve all the groups a user is member of. This command will also take account group membership caused by nested groups. Here’s the first line, it’s a multi-line command that will store all of the groups the users is a member of in the $tokenGroups variable. The groups are represented by their SID.

$tokenGroups = Get-ADUser -SearchScope Base -SearchBase 'CN=thomas,OU=Admin Accounts,DC=contoso,DC=com' `

-LDAPFilter '(objectClass=user)' -Properties tokenGroups | Select-Object `

-ExpandProperty tokenGroups | Select-Object -ExpandProperty Value

In order to easily translate them to their AD AccountName you can use the following command I blogged about earlier (Quick Tip: Resolving an SID to a AccountName)

$groups = $tokengroups | % {((New-Object System.Security.Principal.SecurityIdentifier($_)).Translate( [System.Security.Principal.NTAccount])).Value}

Using the “-SearchSCope Base –SearchBase …” approach seems to be necessary as you cannot simply use Get-ADUser username …

image001

2 comments

Failover Cluster: Generic Applications Fail with OutOfMemoryException

Published on Thursday, August 14, 2014 in , , , , ,

Recently I helped a customer which was having troubles migrating from a Windows 2003 cluster to a Windows 2012 cluster. The resources they were running on the cluster consisted of many in house developed applications. There were about 80 of them and they were running as generic applications.

Due to Windows 2003 being end of life they started a phased migration towards Windows 2012 (in a test environment). At first the migration seemed to go smooth, but at a given moment they were only able to start a limited amount of applications. The applications that failed gave an Out Of Memory exception (OutOfMemoryException). Typically they could start about 25 applications, and from then on they weren’t able to start more. This number wasn’t exact, sometimes it was more, sometimes it was less.

As I suspected that this wasn’t really a failover clustering problem but more a Windows problem I googled for “windows 2012 running many applications out of memory exception”. I found several links:

HP: Unable to Create More than 140 Generic Application Cluster Resources

IBM: Configuring the Windows registry: Increasing the noninteractive desktop heap size

If the parallel engine is installed on a computer that runs Microsoft Windows Server, Standard or Enterprise edition, increase the noninteractive desktop heap size to ensure that a sufficient number of processes can be created and run concurrently

So it seems you can tweak the desktop heap size in the registry. Here is some background regarding the modification we did to the registry.

The key: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

SharedSection has three values: (KB184802: User32.dll or Kernel32.dll fails to initialize)

  • The first SharedSection value (1024) is the shared heap size common to all desktops. This includes the global handle table, which holds handles to windows, menus, icons, cursors, and so forth, and shared system settings. It is unlikely that you would ever need to change this value.
  • The second SharedSection value (3072) is the size of the desktop heap for each desktop that is associated with the "interactive" window station WinSta0. User objects like hooks, menus, strings, and windows consume memory in this desktop heap. It is unlikely that you would ever need to change this second SharedSection value.
  • The third SharedSection value (512) is the size of the desktop heap for each desktop that is associated with a "noninteractive" window station. If this value is not present, the size of the desktop heap for noninteractive window stations will be same as the size specified for interactive window stations (the second SharedSection value).

Default on Windows 2012 seems to be 768

Raising it to 2048 seems to be a workaround/solution. A reboot is required! After this we were able to start up to 200 generic applications (we didn’t test more). However after a while there were some failures, but at first sight quite limited. This might be due to the actual memory being exhausted. Either way, we definitely saw a huge improvement.

Disclaimer: ASKPERF: Sessions, desktops and windows stations

Please do not modify these values on a whim. Changing the second or third value too high can put you in a no-boot situation due to the kernel not being able to allocate memory properly to even get Session 0 set up

Bonus info: why didn’t the customer didn’t have any issues running the same workload on Windows 2003? They configured the generic applications with “allow desktop interaction”. Something which was removed from the generic applications in Windows 2008. Because they had “allow desktop interaction” configured, generic applications were running in an interaction session and thus were not limited by the much smaller non-interactive desktop heap size.

0 comments

SCOM 2012 R2: Web Portal: 503 The Service is Unavailable

Published on Wednesday, August 13, 2014 in ,

The other day one of my customers mentioned that their SCOM Web Portal has been broken for a while now. As I like digging into web application issues I took a quick look. Here’s what I came up with. It seems that the portal itself was loading fine, but viewing All Alerts or Active Alerts showed a Service Unavailable (“HTTP Error 503: The service is unavailable”).

image

One of the things about IIS based errors is that in most cases the Event Log on the web server can help you a great deal. In the System Event Log I found the following:

clip_image001

A process serving application pool 'OperationsManagerMonitoringView' reported a failure trying to read configuration during startup. The process id was '6352'. Please check the Application Event Log for further event messages logged by the worker process on the specific error. The data field contains the error number.

Checking the IIS Management Console I could indeed see that the Application Pool was stopped. Starting it succeeded, but viewing the page made it crash again. Looking a bit further I found the following in the Application Event Log:

clip_image001[5]

The worker process for application pool 'OperationsManagerMonitoringView' encountered an error 'Configuration file is not well-formed XML

' trying to read configuration data from file '\\?\C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG\web.config', line number '14'. The data field contains the error code.

Now that seems pretty descriptive! Using notepad I checked the contents of the file and tried to see why the XML was not well-formed. I checked the XML tags and the closings and such but I couldn’t find anything at first sight. Looking a bit longer I saw that the quotes (“) were different from the other quotes in the file. Here’s a screenshot of the bad line and the fixed line. You can simply erase and retype the “ on that line and you should be good to go.

image

Personally I like taking a backup copy before I perform manual fixes. After saving the file I did an IISReset just to be sure. And after that we were able to successfully view our alerts through the Web Portal again!

4 comments

Static Host Records Disappearing From DNS

Published on in

Somewhere in the past year I started writing the stuff below. I had a specific DNS issue I was looking into. Sadly I never found the real solution, but I found the troubleshooting information interesting enough to save it for future use. The case: there are multiple servers which have one network interface but have multiple IP addresses on them, typically web servers. We prefer them to only register their “primary” IP address in DNS. In order to achieve this we uncheck “Register this connection’s addresses in DNS” and create a static A (and PTR) record for the hostname and the primary IP.

However we are seeing that some of these records seem to disappear after a while.Here’s someone with the same problem: Serverfault.com: Disabling DNS registration on Server 2008 R2

In the end I was able to reproduce this for a given machine:

clip_image002

Enable DNS Client Events logging:

clip_image004

Enable DNS Debug Logging:

clip_image005

And in order to reproduce I made sure both the A and PTR records were gone. Typically A record was disappearing, but PTR remained in place. So I made sure to manually delete the PTR record.

Then we go ahead and create our static record:

clip_image006

And the resulting record:

clip_image007

So in theory, even if scavenging were enabled, it shouldn’t affect this record.

Now we fast forward in time. To be more precise, I found out that the deletion for this specific record was occurring every hour. Using repadmin it’s pretty easy to see when exactly the deletion occurred:

clip_image009

And with AD Auditing enabled:

clip_image010

For more on enabling auditing: Acefekay: DNS Records disappearing and DNS auditing.aspx

So for example at 16:10:22 the DNS debug log shows us the following:

clip_image012

I highlighted the TKEY query as this made me suspect that Dynamic Updates were involved. TKEY is the negotiation for secure updates if I’m not mistaken. You can read more about that there: Technet: DNS Processes and Interactions (Example: How dynamic update works)

On the server I could also see some related events appearing in the DNS Client Events log:

clip_image013

clip_image015

clip_image017

So Dynamic Updates seemed to be removing our statically created records, but it wasn’t active. How is that possible?!

For starters I stumbled upon this KB article: KB2520155: DNS Host record of a computer is deleted after you change the DNS server assignment But that didn’t seem to be applicable as we weren’t touching DNS server settings in any way.

As indicated in the beginning of the explanation there was a post with same problem: Serverfault.com: Disabling DNS registration on server 2008 R2 The solutions provided didn’t seem interesting to me. I tried the netsh command, but it was already set to “none”. Setting a registry key seemed to be a lot of hassle for something which should work right away. I was looking for either a hotfix or a proper way to configure it.

And then I stumbled upon this: Social Technet: Single machine will register all IP addresses in DNS - want to register only one

This led to the following KB articles:

· Vista/ Windows 2008: KB975808: All IP addresses are registered on the DNS servers when the IP addresses are assigned to one network adapter on a computer that is running Windows Server 2008 SP2 or Windows Vista SP2

· Win 7/ Windows 2008 R2: KB2386184: IP addresses are still registered on the DNS servers even if the IP addresses are not used for outgoing traffic on a computer that is running Windows 7 or Windows Server 2008 R2

As far as Win7/ Windows 2008 R2 is included, this hotfix is included in SP1.

After you install this hotfix, you can assign IP addresses that will not be registered for outgoing traffic on the DNS servers by using a new flag of the netsh command. This new flag is the skipassource flag.

>> This makes me wonder if this only affects DNS registration or also “outgoing TCP/IP traffic”: e.g. can we assume that all outgoing traffic will use the primary IP? This would be useful in firewall scenario’s. From what I read here (Technet: Set-NetIPAddress> -SkipAsSource) I think it does.

Sidenote: IP Address selection for outgoing traffic (Blogs.technet.com: Source IP address selection on a Multi-Homed Windows Computer )

The server will use the 192.168.1.68 address because it has the longest matching prefix.

To see this more clearly, consider the IP addresses in binary:

11000000 10101000 00000001 00001110 = 192.168.1.14 (Bits matching the gateway = 25)

11000000 10101000 00000001 01000100 = 192.168.1.68 (Bits matching the gateway = 26)

11000000 10101000 00000001 01111111 = 192.168.1.127 The 192.168.1.68 address has more matching high order bits with the gateway address 192.168.1.127. Therefore, it is used for off-link communication.

In order to use SkipAsSource we have to add additional address from the command line:

· Netsh int ipv4 add address <Interface Name> <ip address> <netmask> skipassource=true

In order to verify this we can execute the following command:

· Netsh int ipv4 show ipaddresses level=verbose

Important remark: there’s also a hotfix for this “feature”: KB2554859: The "skipassource" flag of IP addresses is cleared after you use the GUI to change IP settings of a network adapter in Windows 7 or in Windows Server 2008 R2 It seems that if you use the GUI to modify the “Register this connection’s addresses in DNS” setting it will actually clear the skipassource flag! This hotfix is NOT included in SP1.

Without knowing this already seems to be active for some servers! In fact it seems that Windows Failover Clustering uses this to avoid the “VIPs” to be registered as the hostname.

clip_image018

Contrary to most of my other blog posts this one isn’t as polished or a nice wrap-up, but still to me it has some valuable bits of information regarding DNS troubleshooting as such it goes into my personal archive.

6 comments

Direct Access: No Security Associations

Published on Friday, August 8, 2014 in

I’ve been working on a Direct Access deployment for quite some time now. The clients are Windows 7 SP1 and the DA servers are based on Windows Server 2012. We hand out DA capabilities using a GPO that is scoped to a Windows Active Directory group. Once the computer is in the group, both the DA settings are configured (GPO) and the client requests a computer certificate (auto-enrollment). From this point on, most of the clients have DA connectivity immediately.

Ironically, while creating a document for the helpdesk to be able to resolve basic DA connectivity issues, I ran into the following issue:

netsh dns show state

image

DA is configured and is enabled as to be expected…

ipconfig

image

Our IPHTTPS interface has a valid IP Address. I can even ping the IPv6 address of the DA DNS service (determined by netsh namespace show policy)…

netsh int httpstunnel show int

image

The IPHTTPS interface didn’t show any errors…

netsh advf monitor show mmsa

netsh advf monitor show qmsa

image

Ahah! No SAs match the specified criteria. So no security associations were being made. But… Why?!

After some googling I stumbled upon some threads where they asked is the IKE and AuthIP IPsec Keying Modules service running?

And indeed, on my faulty client this service was stopped and had a startup type of manual. Starting the service just once seemed to immediately reconfigure the startup type to automatic as well. And in a matter of seconds the DA connectivity seemed fine. Security Associations were now successfully created:

image

In the days after more and more newly assigned clients seemed to suffer this phenomena. So, as a workaround, I configured the IKE and AuthIP IPsec Keying Modules service with an automatic startup type using GPO:

image001

And the actual setting:

image002

If anyone has a an explanation as to why this suddenly stopped working I’d be happy to hear it. Perhaps some patch was released who “broke” the DA/IPSec stuff to get the service running?