0 comments

IDX10311: RequireNonce is 'true' (default) but validationContext.Nonce is null

Published on Tuesday, June 14, 2016 in , ,

I’ve been educating myself on the capabilities of OpenID Connect/OAuth in Server 2016. The version I’m currently playing with is based on TP5. I created a small application which consists of a web application and an API. Just for educational purposes. The actual application can be found here: https://github.com/tvuylsteke/TodoListWeb

When I started testing my application I ran into an issue. I would visit my application, hit the sign in button and be redirected to AD FS. I would either enter my credentials or be authenticated transparently and then be redirected to my application. That’s where things went wrong. I always seemed to get this error:

error

In Words:We're having trouble signing you in.

IDX10311: RequireNonce is 'true' (default) but validationContext.Nonce is null. A nonce cannot be validated. If you don't need to check the nonce, set OpenIdConnectProtocolValidator.RequireNonce to 'false'.;

Some online searching led me to some threads but no real good suggestions. I also found a session off Build 2015: Cloud Authentication Troubleshooting and Recipes for Developers They mention that IDX10311 typically happens when you don’t receive an expected cookie from the browser. Likely cause: Your reply URL is sending the browser to somewhere different than where you started. I double checked everything, but that didn’t seem to be the cause.

Now I found out that using chrome everything was working as expected. Still I had no real clue. I posted my issue to an internal DL and one of my colleagues quickly spotted my issue using the Fiddler traces I provided. He told me that the OpenIdConnect.nonce.OpenIdConnect cookie was not being set correctly for the todolistweb.contoso.com application in IE. And when I took my traces I could indeed see this:

A trace from Internet Explorer:

You can see the response from AD FS and then the browser going back to the application without any cookies:

IE1

IE2

Now if we compare that to a session from within Chrome:

Chrome1

You can clearly see the OpenIDConnect.nonce cookie

Chrome2

As a solution to this issue I added my application to the Local Intranet Zone in IE and that resulted in the cookie being sent to the application. Mystery solved!

0 comments

Protected Users Group

Published on Saturday, February 27, 2016 in ,

Earlier this week I’ve been talking to a customer about the “Protected Users” group. You might have seen it appearing when introducing the first 2012 R2 domain controller. Here’s a good explanation on its purpose:

Protected Users is a new global security group to which you can add new or existing users. Windows 8.1 devices and Windows Server 2012 R2 hosts have special behavior with members of this group to provide better protection against credential theft. For a member of the group, a Windows 8.1 device or a Windows Server 2012 R2 host does not cache credentials that are not supported for Protected Users. Members of this group have no additional protection if they are logged on to a device that runs a version of Windows earlier than Windows 8.1. Source: TechNet: How to Configure Protected Accounts

The above is actually a bit misleading. The functionality was actually backported to Windows 2008 R2/Windows 2012 in the hotfix KB2871997 See blogs.technet.com: An Overview of KB2871997 for an explanation on this.

This group might be part of your organization’s strategy to reduce the attack surface for pass the hash. A great white paper on this can be found here: Mitigating Pass-the-Hash (PtH) Attacks and Other Credential Theft, Version 1 and 2

One of the things the Protected Users group ensures is that no NTLM hashes are available to be used or stolen. Now I wanted to see this for myself. There are various tools out there that are capable of listing the various secrets. I tried Windows Credential Editor (WCE) but that one didn’t work on (my) Windows 2012 R2. So I used Mimikatz. My setup: A 2012 R2 domain controller and a 2012 R2 member server. I’ve got 3 domain admins: one that has the remote desktop session open to the member server and then two that have a powershell runnning through runas. Of the latter one is a member of the Protected Users group:

Run as different user: SETSPN\john

image

Run as different user: SETSPN\thomas

image

As you can see John is an oldschool Domain Admin whereas Thomas has read the Mitigating PtH whitepaper and is a proud member of the Protected Users group. This is the PowerShell oneliner I used to dump the groups I care about: WHOAMI /GROUPS /FO CSV | ConvertFrom-Csv | where {$_."group name" -like "Setspn\*"}

Here you can see the Protected Users admin has no NTLM available:

image

Where the regular admin has NTLM available:

image

Here’s the difference from an attacker point of view:

Start Mimikatz –> Privilege::debug –> sekurlsa::logonpasswords And here are the goodies:

John:

Authentication Id : 0 ; 3529276 (00000000:0035da3c)
Session           : Interactive from 0
User Name         : john
Domain            : SETSPN
Logon Server      : SRVDC01
Logon Time        : 2/24/2016 6:59:54 PM
SID               : S-1-5-21-4274776166-1111691548-620639307-5603
        msv :
         [00000003] Primary
         * Username : john
         * Domain   : SETSPN
         * NTLM     : 59884edfb057d0fec8cb7e0d571dc200
         * SHA1     : 7e655db2b3a7e88fb0c50ca56416ae655469f09e
         [00010000] CredentialKeys
         * NTLM     : 59884edfb057d0fec8cb7e0d571dc200
         * SHA1     : 7e655db2b3a7e88fb0c50ca56416ae655469f09e
        tspkg :
        wdigest :
         * Username : john
         * Domain   : SETSPN
         * Password : (null)
        kerberos :
         * Username : john
         * Domain   : SETSPN.LOCAL
         * Password : (null)
        ssp :
        credman :

Thomas:

Authentication Id : 0 ; 3493146 (00000000:00354d1a)
Session           : Interactive from 0
User Name         : thomas
Domain            : SETSPN
Logon Server      : SRVDC01
Logon Time        : 2/24/2016 6:59:36 PM
SID               : S-1-5-21-4274776166-1111691548-620639307-5602
        msv :
         [00010000] CredentialKeys
         * RootKey  : db1c2347608db0c4e2d89bbd6c328bf6f42671b7d88653cd4cc9af2713
e958f0
         * DPAPI    : 63adfe49948fca81c885933b3aa23eba
        tspkg :
        wdigest :
         * Username : thomas
         * Domain   : SETSPN
         * Password : (null)
        kerberos :
         * Username : thomas
         * Domain   : SETSPN.LOCAL
         * Password : (null)
        ssp :
        credman :

As you can see the admin that’s a member of the Protected Users group does NOT have the NTLM hashes dumped. Wooptiedoo! Now think and test before you start adding the Domain Admins group to the Protected Users group! By no means you should do that! Here’s some good information on how to start with the Protected Users group and some additional caveats: How to Configure Protected Accounts

Here’s one from my side: after adding my admin user to the Protected Users group he was no longer to RDP to a 2012 R2 member server:

image 

In words: A user account restriction (for example, a time-of-day restriction) is preventing you from logging on. For assistance, contact your system administrator or technical support.

Remote desktop to a Windows 2008 R2 worked fine with that account. It seems for my Protected User admin to be able to log on to a Windows 2012 R2 server it had to actualy use mstsc.exe /restrictedadmin and I had to enable Restricted Admin mode on the member server:

image

You can find that value below HKLM\SYSTEM\CurrentControlSet\Control\Lsa

If you want to know more about the Protected Users group and the Restricted Admin feature read up on both of them here: TechNet: Credentials Protection and Management or digital-forensics.sans.org:Protecting Privileged Domain Accounts: Restricted Admin and Protected Users

Some additional reading on Restricted Admin mode: Restricted Admin mode for RDP in Windows 8.1 / 2012 R2

6 comments

Direct Access: Windows Internal Database (SQL) High CPU Usage

Published on Wednesday, October 14, 2015 in ,

[Update 2016/03/15] An official article has been released with proper guidance and no need for any SQL management tools at all. It’s available here: https://technet.microsoft.com/en-us/library/mt693376.aspx

I’ve got a customer who has deployed Direct Access quite a while ago. Something which we have observed for a while now is that the CPU usage of the servers is rather high. Some details about our setup: we got 2 Direct Access servers which are load balanced using Windows NLB. They are running Windows 2012 R2, have 4 vCPU’s and 8 GB of RAM. When troubleshooting this issue, we were seeing 400 active users, roughly 200 for each server. Here’s what the CPU usage looked like:

image001

As you can see sqlservr.exe is using 67% CPU. Now that’s quite a lot… I would hope a DA server had other things to do with it’s CPU instead of running an SQL instance. Now I know where this instance comes from. We configured inbox accounting on the Direct Access servers. This allows an administrator to pull up reports about who connected when to what resources. You can choose between Radius and Windows Internal Database (WID) for the auditing data targets. We choose the WID approach. We configured our accounting to hold data for 3 months. So I started wondering, is the SQL database instance having troubles with the amount of data? Or is there an issue with indexes that are fragmented or… In order to investigate this, we’d had to do some SQL talking to this instance. As it’s a WID instance, we can only talk to it from the box itself. So we can either install the SQL commandline tools or the SQL Management Studio. I’m not an SQL guru, so I prefer to do my troubleshooting using the SQL Management Studio. In order to determine what version you can use you can check the location for the sqlservr.exe binary:

image003

And from the details you can see that a WID on a Windows 2012 R2 is actually build 11.0.2100.60 which, if bing is correct, is a SQL 2012 edition.

image005

So I took the SQL 2012 iso and installed the SQL Management Studio on the DA servers. Watch out when going through the setup, we don’t want to install another SQL instance! Just the management tools. Here’s the string we can use to connect to the instance: \\.\pipe\MICROSOFT##WID\tsql\query

image007

After connecting to the instance we see that there’s only one Database (RaAcctDb) which has 4 tables. This query I found here: TechNet Gallery and also resembles the query that is presented here: KB2755960. To check all indexes for fragmentation issues, execute the following query:

SELECT OBJECT_NAME(ind.OBJECT_ID) AS TableName,

ind.name AS IndexName, indexstats.index_type_desc AS IndexType,

indexstats.avg_fragmentation_in_percent

FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, NULL) indexstats

INNER JOIN sys.indexes ind

ON ind.object_id = indexstats.object_id

AND ind.index_id = indexstats.index_id

WHERE indexstats.avg_fragmentation_in_percent > 30

ORDER BY indexstats.avg_fragmentation_in_percent DESC

No indexes were returned. Another thing which I hear from time to time is rebuild “statistics”. So I checked them and I saw they were two weeks old. I figured rebuilding them couldn’t hurt:

 

use RaAcctDb;

UPDATE STATISTICS connectionTable

WITH FULLSCAN

GO

 

use RaAcctDb;

UPDATE STATISTICS EndpointsAccessedTable

WITH FULLSCAN

GO

 

use RaAcctDb;

UPDATE STATISTICS ServerEndpointTable

WITH FULLSCAN

GO

 

use RaAcctDb;

UPDATE STATISTICS SessionTable

WITH FULLSCAN

GO

Again no real change in CPU usage… Ok, back to the drawing board.  I googled a bit for “high cpu usage SQL” and I found the following blog: http://mssqlfun.com/2013/04/01/dmv-3-what-is-currently-going-on-sys-dm_exec_requests-2/ One of the queries there is this one:

 

SELECT

R.SESSION_ID,

R.REQUEST_ID AS SESSION_REQUEST_ID,

R.STATUS,

S.HOST_NAME,

C.CLIENT_NET_ADDRESS,

CASE WHEN S.LOGIN_NAME = S.ORIGINAL_LOGIN_NAME THEN S.LOGIN_NAME ELSE S.LOGIN_NAME + '(' + S.ORIGINAL_LOGIN_NAME + ')' END AS LOGIN_NAME,

S.PROGRAM_NAME,

DB_NAME(R.DATABASE_ID) AS DATABASE_NAME,

R.COMMAND,

ST.TEXT AS QUERY_TEXT,

QP.QUERY_PLAN AS XML_QUERY_PLAN,

R.WAIT_TYPE AS CURRENT_WAIT_TYPE,

R.LAST_WAIT_TYPE,

R.BLOCKING_SESSION_ID,

R.ROW_COUNT,

R.GRANTED_QUERY_MEMORY,

R.OPEN_TRANSACTION_COUNT,

R.USER_ID,

R.PERCENT_COMPLETE,

CASE R.TRANSACTION_ISOLATION_LEVEL

WHEN 0 THEN 'UNSPECIFIED'

WHEN 1 THEN 'READUNCOMITTED'

WHEN 2 THEN 'READCOMMITTED'

WHEN 3 THEN 'REPEATABLE'

WHEN 4 THEN 'SERIALIZABLE'

WHEN 5 THEN 'SNAPSHOT'

ELSE CAST(R.TRANSACTION_ISOLATION_LEVEL AS VARCHAR(32))

END AS TRANSACTION_ISOLATION_LEVEL_NAME

FROM

SYS.DM_EXEC_REQUESTS R

LEFT OUTER JOIN SYS.DM_EXEC_SESSIONS S ON S.SESSION_ID = R.SESSION_ID

LEFT OUTER JOIN SYS.DM_EXEC_CONNECTIONS C ON C.CONNECTION_ID = R.CONNECTION_ID

CROSS APPLY SYS.DM_EXEC_SQL_TEXT(R.SQL_HANDLE) ST

CROSS APPLY SYS.DM_EXEC_QUERY_PLAN(R.PLAN_HANDLE) QP

WHERE

R.STATUS NOT IN ('BACKGROUND','SLEEPING')

The result:

image009

It returns one ore more queries the SQL instance is currently working on. It’s actually pretty easy and very powerful. The first record is a sample entry we care about. The others are me interacting with the SQL management studio. Scroll to the right and you’ll see both the execution plan and the actual query. Now how cool is that?!

image011

There we can get the query being executed (QUERY_TEXT)

CREATE PROCEDURE raacct_InsertSession (

            @Hostname NVARCHAR(256),

            @ClientIPv4Address BINARY(4),

            @ClientIPv6Address BINARY(16),

            @ClientISPAddressType SMALLINT,

            @ClientISPAddress VARBINARY(16),

            @ConnectionType TINYINT,

            @TransitionTechnology INT,

            @TunnelType INT,

            @SessionHandle BIGINT,

            @Username NVARCHAR(256),

            @SessionStartTime BIGINT,

            @AuthMethod INT,

            @HealthStatus INT) AS

BEGIN

    DECLARE @SessionId BIGINT

    DECLARE @ConnectionId BIGINT

    DECLARE @NumActiveSessions SMALLINT

    IF (@SessionHandle IS NULL OR @SessionHandle = 0)

    BEGIN

        -- error (BAD PARAMETER)

        RETURN (1)

    END

    IF (@SessionStartTime IS NULL OR @SessionStartTime = 0)

    BEGIN

        -- error (BAD PARAMETER)

        RETURN (1)

    END

    SELECT @SessionId = 0

    BEGIN TRANSACTION

    SELECT @SessionId = [SessionId]

    FROM [dbo].[SessionTable]

    WHERE    @SessionHandle = [SessionHandle]

        AND @SessionStartTime = [SessionStartTime]

 

    IF (@@ROWCOUNT > 0)

    BEGIN

        -- error (session already exists)

        ROLLBACK TRANSACTION

        RETURN (2)

    END

    -- check if connection exists

    SELECT @ConnectionId = connTbl.[ConnectionId]

    FROM [dbo].[ConnectionTable] AS connTbl, [dbo].[SessionTable] AS sessTbl

    WHERE sessTbl.SessionState = 1

      AND connTbl.ConnectionId = sessTbl.ConnectionId

      AND connTbl.Hostname = @Hostname

      AND connTbl.ClientIPv4Address = @ClientIPv4Address

      AND connTbl.ClientIPv6Address = @ClientIPv6Address

      AND connTbl.ClientISPAddressType = @ClientISPAddressType

      AND connTbl.ClientISPAddress = @ClientISPAddress

      AND connTbl.ConnectionType = @ConnectionType

      AND connTbl.TransitionTechnology = @TransitionTechnology

      AND connTbl.TunnelType = @TunnelType

    IF @@ROWCOUNT = 0

    BEGIN

        -- create connection record

        INSERT INTO [dbo].[ConnectionTable] ([Hostname],

                [ClientIPv4Address],

                [ClientIPv6Address],

                [ClientISPAddressType],

                [ClientISPAddress],

                [ConnectionType],

                [TransitionTechnology],

                [TunnelType]

                )

            VALUES (@Hostname,

            @ClientIPv4Address,

            @ClientIPv6Address,

            @ClientISPAddressType,

            @ClientISPAddress,

            @ConnectionType,

            @TransitionTechnology,

            @TunnelType

            )

        IF @@ERROR <> 0

        BEGIN

            -- error (failed to create connection), return from here

            ROLLBACK TRANSACTION

            RETURN (99)

        END

        SET @ConnectionId = @@IDENTITY

    END

    SELECT @NumActiveSessions = COUNT(SessionHandle)

    FROM [dbo].[SessionTable]

    WHERE   [SessionState] = 1

    SET @NumActiveSessions = @NumActiveSessions + 1

    INSERT INTO [dbo].[SessionTable] ([ConnectionId],

                [SessionHandle],

                [Username],

                [SessionStartTime],

                [AuthMethod],

                [HealthStatus],

                        [NumConcurrentConnections]

                )

        VALUES (@ConnectionId,

            @SessionHandle,

            @Username,

                  @SessionStartTime,

            @AuthMethod,

            @HealthStatus,

            @NumActiveSessions

        )

    IF @@ERROR <> 0

    BEGIN

        ROLLBACK TRANSACTION

        RETURN (4)

    END

    COMMIT TRANSACTION

END

 

The only thing I can see, with my limited SQL knowledge, is that potentially performance hits might occur on the where statements:

 

FROM [dbo].[SessionTable]

    WHERE    @SessionHandle = [SessionHandle]

        AND @SessionStartTime = [SessionStartTime]

And

FROM [dbo].[ConnectionTable] AS connTbl, [dbo].[SessionTable] AS sessTbl

    WHERE sessTbl.SessionState = 1

      AND connTbl.ConnectionId = sessTbl.ConnectionId

      AND connTbl.Hostname = @Hostname

      AND connTbl.ClientIPv4Address = @ClientIPv4Address

      AND connTbl.ClientIPv6Address = @ClientIPv6Address

      AND connTbl.ClientISPAddressType = @ClientISPAddressType

      AND connTbl.ClientISPAddress = @ClientISPAddress

      AND connTbl.ConnectionType = @ConnectionType

      AND connTbl.TransitionTechnology = @TransitionTechnology

      AND connTbl.TunnelType = @TunnelType

The where statements act as filters and columns they use are often indexed. Without an index the SQL server would have to scan the complete table looking for the records. Now on smaller tables that’s not an issue but the SessionTable table contains 14.482.972 records!

image013

So if we check the indexes for that table, one would hope SessionHandle, SessionStartTime and SessionState to be present:

image015

The last one UQ_SessionT… seems to have both SessionHandle and SessionStartTime in it. So I guess that should satisfy the first where statement:

image017

Now what about SessionState? I can’t seem to find that one… Now back to our query that showed us the query being executed. There’s also an XML_QUERY_PLAN. It’s clickable in the Management Studio:

image019

See how this query cost shows 50%? Further down there’s another Query that shows the other 50%. Both show “missing index”:

image021

As previously stated, I’m not an experienced SQL engineer/DBA. I try to crosscheck stuff I find online before applying it. Also I wouldn’t do this kind of stuff on a FIM Service or SCCM database. Those are pretty complex databases. But I made a personal assessment and the Direct Access auditing database seems simple enough to tinker with it. So I decided to give it a try and create the index. Undoing this is pretty straightforward, so I guess there’s no real harm in going forward. Right-click one of the existing indexes and choose Script Index as > CREATE To > New Query Editor Window

image023

Simply change both the Index name and the column to “SessionState”. And execute the query. After refreshing the UI you can see the index:

image025

And there goes the CPU usage:

image027

Conclusion: to me it looks like the DA team just forgot this particular index. From the other indexes you can tell they actually did something for those. I’m not really sure why we didn’t just log a case with Microsoft. Partially I guess because we were afraid/guessing we’d get the answer: by design with that amount of auditing data. But after this troubleshooting session we can clearly see there’s shortcoming in the SQL database setup. As with most stuff you read on the internet: be careful when applying in your environment. If you do not know what commands/queries you’re executing, look them up and do some background reading.