Somewhere in the past year I started writing the stuff below. I had a specific DNS issue I was looking into. Sadly I never found the real solution, but I found the troubleshooting information interesting enough to save it for future use. The case: there are multiple servers which have one network interface but have multiple IP addresses on them, typically web servers. We prefer them to only register their “primary” IP address in DNS. In order to achieve this we uncheck “Register this connection’s addresses in DNS” and create a static A (and PTR) record for the hostname and the primary IP.

However we are seeing that some of these records seem to disappear after a while.Here’s someone with the same problem: Serverfault.com: Disabling DNS registration on Server 2008 R2

In the end I was able to reproduce this for a given machine:

Enable DNS Client Events logging:

Enable DNS Debug Logging:

And in order to reproduce I made sure both the A and PTR records were gone. Typically A record was disappearing, but PTR remained in place. So I made sure to manually delete the PTR record.

Then we go ahead and create our static record:

And the resulting record:

So in theory, even if scavenging were enabled, it shouldn’t affect this record.

Now we fast forward in time. To be more precise, I found out that the deletion for this specific record was occurring every hour. Using repadmin it’s pretty easy to see when exactly the deletion occurred:

And with AD Auditing enabled:

For more on enabling auditing: Acefekay: DNS Records disappearing and DNS auditing.aspx

So for example at 16:10:22 the DNS debug log shows us the following:

I highlighted the TKEY query as this made me suspect that Dynamic Updates were involved. TKEY is the negotiation for secure updates if I’m not mistaken. You can read more about that there: Technet: DNS Processes and Interactions (Example: How dynamic update works)

On the server I could also see some related events appearing in the DNS Client Events log:

So Dynamic Updates seemed to be removing our statically created records, but it wasn’t active. How is that possible?!

For starters I stumbled upon this KB article: KB2520155: DNS Host record of a computer is deleted after you change the DNS server assignment But that didn’t seem to be applicable as we weren’t touching DNS server settings in any way.

As indicated in the beginning of the explanation there was a post with same problem: Serverfault.com: Disabling DNS registration on server 2008 R2 The solutions provided didn’t seem interesting to me. I tried the netsh command, but it was already set to “none”. Setting a registry key seemed to be a lot of hassle for something which should work right away. I was looking for either a hotfix or a proper way to configure it.

And then I stumbled upon this: Social Technet: Single machine will register all IP addresses in DNS - want to register only one

This led to the following KB articles:

· Vista/ Windows 2008: KB975808: All IP addresses are registered on the DNS servers when the IP addresses are assigned to one network adapter on a computer that is running Windows Server 2008 SP2 or Windows Vista SP2

· Win 7/ Windows 2008 R2: KB2386184: IP addresses are still registered on the DNS servers even if the IP addresses are not used for outgoing traffic on a computer that is running Windows 7 or Windows Server 2008 R2

As far as Win7/ Windows 2008 R2 is included, this hotfix is included in SP1.

After you install this hotfix, you can assign IP addresses that will not be registered for outgoing traffic on the DNS servers by using a new flag of the netsh command. This new flag is the skipassource flag.

>> This makes me wonder if this only affects DNS registration or also “outgoing TCP/IP traffic”: e.g. can we assume that all outgoing traffic will use the primary IP? This would be useful in firewall scenario’s. From what I read here (Technet: Set-NetIPAddress> -SkipAsSource) I think it does.

Sidenote: IP Address selection for outgoing traffic (Blogs.technet.com: Source IP address selection on a Multi-Homed Windows Computer )

The server will use the 192.168.1.68 address because it has the longest matching prefix.

To see this more clearly, consider the IP addresses in binary:

11000000 10101000 00000001 00001110 = 192.168.1.14 (Bits matching the gateway = 25)

11000000 10101000 00000001 01000100 = 192.168.1.68 (Bits matching the gateway = 26)

11000000 10101000 00000001 01111111 = 192.168.1.127 The 192.168.1.68 address has more matching high order bits with the gateway address 192.168.1.127. Therefore, it is used for off-link communication.

In order to use SkipAsSource we have to add additional address from the command line:

· Netsh int ipv4 add address <Interface Name> <ip address> <netmask> skipassource=true

In order to verify this we can execute the following command:

· Netsh int ipv4 show ipaddresses level=verbose

Important remark: there’s also a hotfix for this “feature”: KB2554859: The "skipassource" flag of IP addresses is cleared after you use the GUI to change IP settings of a network adapter in Windows 7 or in Windows Server 2008 R2 It seems that if you use the GUI to modify the “Register this connection’s addresses in DNS” setting it will actually clear the skipassource flag! This hotfix is NOT included in SP1.

Without knowing this already seems to be active for some servers! In fact it seems that Windows Failover Clustering uses this to avoid the “VIPs” to be registered as the hostname.

Contrary to most of my other blog posts this one isn’t as polished or a nice wrap-up, but still to me it has some valuable bits of information regarding DNS troubleshooting as such it goes into my personal archive.

I’ve been working on a Direct Access deployment for quite some time now. The clients are Windows 7 SP1 and the DA servers are based on Windows Server 2012. We hand out DA capabilities using a GPO that is scoped to a Windows Active Directory group. Once the computer is in the group, both the DA settings are configured (GPO) and the client requests a computer certificate (auto-enrollment). From this point on, most of the clients have DA connectivity immediately.

Ironically, while creating a document for the helpdesk to be able to resolve basic DA connectivity issues, I ran into the following issue:

netsh dns show state

DA is configured and is enabled as to be expected…

ipconfig

Our IPHTTPS interface has a valid IP Address. I can even ping the IPv6 address of the DA DNS service (determined by netsh namespace show policy)…

netsh int httpstunnel show int

The IPHTTPS interface didn’t show any errors…

netsh advf monitor show mmsa

netsh advf monitor show qmsa

Ahah! No SAs match the specified criteria. So no security associations were being made. But… Why?!

After some googling I stumbled upon some threads where they asked is the IKE and AuthIP IPsec Keying Modules service running?

And indeed, on my faulty client this service was stopped and had a startup type of manual. Starting the service just once seemed to immediately reconfigure the startup type to automatic as well. And in a matter of seconds the DA connectivity seemed fine. Security Associations were now successfully created:

In the days after more and more newly assigned clients seemed to suffer this phenomena. So, as a workaround, I configured the IKE and AuthIP IPsec Keying Modules service with an automatic startup type using GPO:

And the actual setting:

If anyone has a an explanation as to why this suddenly stopped working I’d be happy to hear it. Perhaps some patch was released who “broke” the DA/IPSec stuff to get the service running?

[Update 2014-04-17] Thanks to Steve’s comment I learned that the HEX notation is absolutely not a must. You can just use the IP address but unlike simple filters like Destination or Source you must not use quotes around the IP! Using quotes for the IP will give you a valid filter but no matches will be found. So there’s absolutely no benefit in using the HEX notation. Makes by post a bit useless, but at least I learned something out of it!

In the past I often used Wireshark to debug all kinds of issues. The last year I’ve been using Microsoft Network Monitor 3.4 more and more. I don’t think Network Monitor is better or worse than Wireshark, but Network Monitor has the capabilities to use a trace file generated by the built-in tracing engine of Windows ( See Network Tracing Awesomeness) That means I don’t have to install Wireshark allover the place!

Small side note: Network Monitor 3.4 has been out for a while and I’ve often wondered when a newer version would be released. I wasn’t missing any particular features, but hey I’m an IT guy, I like new stuff! And today I happen to stumble upon this: http://blogs.technet.com/b/messageanalyzer/ Seems like Microsoft does have a successor: Message Analyzer. I installed it an did some quick tests and it seems like there’s a lot of fancy stuff in there! Using it will take some time to learn though. Some fancy features: graphs (e.g. SMB performance) and remote live capturing!!! Awesome!

Either way, back to my good old Network Monitor. One of the things I often do is blindly capture everything and then I try to filter the data that is displayed. One thing I often require is to ensure only traffic with a particular host is displayed. Typically I did this by adding a filter which is of the format “Source == IP and Destination == IP”. This does work mostly, but sometimes it doesn’t when the IP is translated to the actual DNS name. Besides that, it always bothered me I had to type such a long filter.

Next to “Source” and “Destination” there’s also IPv4.SourceAddress, IPv4.DestinationAddress and IPv4.Address. I tried those in the past a few time: e.g. IPv4.Address == “10.6.69.121” and although the filter is accepted, no traffic is shown. Today I accidentally found out that those IPv4.XYZ filters expect an IP in HEX format!

If you got some captured data and you want to filter you can simply drill down the IP packet information, right-click sourceAddress and choose add to display filter. That will give you the hex notation.

Now this works perfectly if you want to do it for a display filter. But what if you want to limit the amount of data captured by using a capture filter?

Sample filter:

Well you can just use a simple IP to HEX converter site like this: http://www.miniwebtool.com/ip-address-to-hex-converter/?ip=10.6.69.121

Or you can use the following PowerShell oneliner:

"$(([Net.IPAddress]"10.6.69.121").GetAddressBytes() | ForEach-Object { '{0:x2}' -f $_ })" -Replace '\s'

The PowerShell oneliner will require you to add an x after the first 0 though.

Happy tracing!

Some weeks ago I had a customer which experienced issues with RemoteApp on Windows 8: Setspn: Windows 8.1 and the RemoteApp Connection URL GPO Setting Issue Now I had another customer with a similar issue. This one was working with Windows 7 and we were entering the URL manually (for now). Upon entering the URL we got greeted with an error:

In order to find out what was going on I thought I’d run a process monitor alongside with it. After filtering most of the success/harmless messages, here’s what’s left:

The one that got my attention was the “NOT IMPLEMENTED” one. It got me thinking. The users logging on to Windows 7 have their appDataRoaming redirected to a folder in their home share. And we use offline files and folders as well. Due to the fact that I was messing around with Direct Access I had my home folder as "work offline”:

After transitioning to “work online”:

All seemed to go well:

Seems a pretty specific case, but as usual, if I ran into it, perhaps other people will as well.

Recently I had to generate a request file for a SAN (Subject Alternative Name) certificate. Using the GUI this is pretty straight forward, but I wanted to use the command line as this allows to be repeated for other certificates way faster. The tool to be used, which is installed by default on Windows, is certreq.exe. Typically certreq.exe uses an inf file to gather most of the input. For the actual parameters I started googling around. I quickly stumbled upon: KB931351: How to add a subject alternative name to a secure LDAP certificate

The relevant section:

[RequestAttributes]
SAN="dns=name.contoso.com&dns=othername.contoso.com"

The generation of the request file went flawless with this parameter. However upon verification using an online CSR decoder (https://www.sslshopper.com/csr-decoder.html ) I couldn’t find my SAN attribute. So I googled a bit more and finally came up with the following contents for certreq.ini:

[Version]

Signature="$Windows NT$"

[NewRequest]
Subject = "CN=name.contoso.com,O=CSR Demo,OU=IT,L=Brussels,S=Brussels,E=certificates@contoso.com,C=BE"

;EncipherOnly = FALSE
Exportable = TRUE   ; TRUE = Private key is exportable
KeyLength = 2048     ; Valid key sizes: 1024, 2048, 4096, 8192, 16384
KeySpec = 1          ; Key Exchange – Required for encryption
KeyUsage = 0xA0      ; Digital Signature, Key Encipherment
MachineKeySet = True
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"

RequestType = PKCS10 ; or CMC.

[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.1 ; Server Authentication
;OID=1.3.6.1.5.5.7.3.2 ; Client Authentication  // Uncomment if you need a mutual TLS authentication

[Extensions]
2.5.29.17 = "{text}"
_continue_ = "dns=name.contoso.com&"
_continue_ = "dns=othername.contoso.com"

As you can see the SAN properties are now specified in an other way, and these seem to make it to the certificate request. I also highlighted Exportable (Private Key) = TRUE but that’s entirely personally and dependent on your scenario. To conclude the parameter required to actually perform the procedure:

• Generate the request file: Certreq.exe –new certreq.ini certreq.req
• Accept and Install certificate: Certreq.exe –accept certificate.cer

Recently I worked together with some Microsoft consultants to implement a Direct Access proof of concept. Initially the configuration went just fine. However, the operational state page showed that we had an issue with the IPSEC certificate. Initially we thought the problem had to do with our DA server being unable to reach the internet in order to download the CRL (Certificate Revocation List). So we had two options: either ask the network team to open up the DA’s external interface connectivity towards the internet or configure the DA server so it could use a proxy.

Due to the workload of the network team, not the best reasoning though, we choose option two: the proxy. We configured the proxy in IE and we were able to successfully download the CRL file. The error didn’t go away though. So we figured we had to configure the system to use the proxy. This can be done using netsh:

• netsh winhttp set proxy 10.0.10.30:8080

The error didn’t went away and it was time for lunch so we thought we’d give the DA server a reboot and some time. When we came back there was something weird with the operation state page. It just said Name_Of_DA_Server is not working properly. We tried rerunning the configuration wizard but we got greeted with the following error:

In words: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. Suddenly we figured out that our netsh winhttp proxy might have broken things a bit. So we gave the command a little twist:

• netsh winhttp set proxy 10.0.10.30:8080 bypass-list="<local>;*.contoso.local"

Now that we used this command our DA console was back to normal, but our IPSEC certificate was still in error. The solution was actually quite simple: just make sure the DA server also has a certificate based upon the Computer (or your custom computer) template. Such a certificate can be used for both client and server authentication. This certificate should be issued by the same CA that issues certificates for your clients and that is configured in the DA configuration wizard. After enrolling a certificate we now had a working DA setup. We could see our clients setting up a connection successfully.

One thing that bothered us though: on the operation state page we had an error on the availability of the IPHTTPS interface of the DA server. Now that was odd. netsh showed it as available and after all we had several clients connected to it. When we removed the proxy (netsh winhttp reset proxy) and then the status page went all green!

So for now I’m not even sure if the DA server needs to check the CRL’s of the certificate used by IPHTTPS. But I guess we’ll find out soon if DA stops working. All I can say: the netsh winhttp set proxy doesn’t seems a really good idea to use. Things started breaking in weird and unpredictable ways.

I’m not sure whether there will be another person ever running into this, but if you do I hope this saves you some time.

One of my customers is deploying Windows 8.1 clients. They mainly use SCCM and App-V as a software delivery solution. Besides that they also have some applications published over RemoteApp. Windows 8 has a GPO setting which allows you to configure the RemoteApp Connection URL: Setting the default RemoteApp connection URL on your clients using GPO

My customer mentioned to me that some of the clients got the following error message: There are currently no connections available on this computer.

Verifying the RemoteApp and Desktop Connections event log showed no entries. I double checked that the GPO was applying by checking the following registry key:

• Key: HKEY_CURRENT_USER\Software\Policies\Microsoft\Workspaces
• Value: DefaultConnectionURL: https://termsrv.contoso.local/rdweb/Feed/webfeed.aspx

I did some googling and stumbled upon the following articles:

• RemoteApp Default Connection URL Group Policy not working
• Default Connection URL via GPO 2012 server / windows 8 client

The first one didn’t apply for our situation. Our clients were not part of any VDI or TS setup. The second one seemed interesting. Sadly no solution was given, but it gave us the hint that adding the user to the local administrators might help. And indeed this made the RemoteApp Connection URL work. If adding a user to the local administrator is a solution for any given problem, this means you got to tweak one of the below:

• Permissions in the registry
• Permissions on the file subsystem
• User Rights Assignment (GPO/Secpol.msc)

I removed my user from the Local Administrators and started testing again. Using process monitor I couldn’t see any particular access problems for either file or registry. What I did learn from process monitor though is that a part of the RemoteApp configuration is handled by a group of scheduled tasks! I could clearly see these being created as an Administrator, but not for my regular account. Here’s how they should look:

So somehow the creation of the scheduled tasks went sideways. I thought I ‘d be smart and I started with enabling all failure audits for all events in the security event log: auditpol /set /category:* /failure:enable. Whilst I could see some events being generated during my logon, I couldn’t relate any of those to the scheduled tasks not being created. So no clues there. The only thing I figured I could do is start examining the user right assignment settings. This customer has implemented the security recommendations (Security Compliancy Manager) which means a lot of these are customized. After I put my client in an OU with NO GPO’s whatsoever I could see that my user got this RemoteApp configured just fine.

I thought this would be an easy job. I started comparing my user right assignments on the client (with no GPO’s) and the ones being set by GPO. Typically I would need to hunt something down where by default users have the right, but where the GPO is more strict and only Administrators have it. After going through the list I had found none of those…

So I cloned the GPO, made sure the original one no longer applied to my client and started setting settings to not configured in the user rights assignment section. Sadly I started at the bottom because the one which seemed to be the culprit is Act as part of the operating system. The GPO seemed to set grant this right to “Authenticated Users”. That made me frown… It seems like a very privileged thingy to grant to authenticated users… From the SCM 3.0 toolkit you can see that both the default and the advised setting is “None”:

It seems that somehow the GPO got wrongly configured at some point in time. By default this is set to “None”. After removing the authenticated users from this particular setting I was finally able to get my user his RemoteApp configuration up and running:

This is the setting which seemed to be having this negative effect:

I ‘m still trying to wrap my head around the fact that whilst this setting had authenticated users in it, only non-administrator users were impacted. Either way, case closed!