A while ago I had to publish Exchange 2010 services across TMG 2010. All what’s below is more or less from the TMG administrator point of view. All of the tests are done externally (across TMG).

Also, important to note: my customer had an e-mail domain which was different from the Active Directory UPN. E.g.

• thomas@customer.local (AD UPN)
• thomas@customer.com (E-mail domain)

I believe this information will be important as a lot of the autodiscover GUI pieces only ask for an e-mail address and a password. There’s no room to provide the username. Before diving into Autodiscover & Outlook Anywhere I’m going to go into the basics of publishing Exchange Services across TMG.

For starters, in a typical scenario where all Exchange Services are published there are 3 publishing rules. Each are listed with the defaults paths published as the TMG wizards configure them.

1. Outlook Web Access (OWA)

2. Outlook Anywhere (with support for Outlook 2007 clients):

3. Active Sync

Each of these paths points to a vdir on the Exchange CAS server. Here’s a screenshot of all available vdir’s:

TMG Concepts: The Listener

Now obviously when you are publishing applications there’s got to come some authentication in play. On the internet side of things there’s one or more listeners. A listener determines which kind of authentication method the user on the internet is challenged with. For instance we have Form Based Authentication. In form based authentication the user is present with a nice TMG logo’d form and is asked to provide credentials. So to summarize, as listener is a way for TMG to capture the end user his credentials in a given form.

TMG Concepts: The Publishing Rule

On the other hand we got our applications on the intranet which require credentials to be presented in each request in order to show the appropriate content. This kind of behavior is determined in the publishing rule of each service. One of the items we configure in the publishing rule is the authentication delegation. We have to specify the way TMG can use the credentials it gathered on the listener to authenticate to the service in the backend.

And this is where troubles can begin. You always have to match the authentication delegation settings with the authentication protocols enabled on the vdirs. For instance if you configure the OWA website in Exchange to be available with basic authentication, there’s no point in configuring TMG to publish it with NTLM or Kerberos. The “Test Rule” button in TMG will show you this right away. I often encounter the missmatch where the Exchange Admin configured forms-based authentication on Exchange for OWA. TMG can’t handle this, unless you allow unauthenticated traffic to the Exchange server…. Now if you are an IIS savvy admin you could start tweaking IIS right away from the IIS management console. I would advise against this. All of the IIS configuration for Exchange related web services can be performed to the Exchange Management Shell or Exchange Management Console. Here’s a screenshot for OWA from withing the EMC.

Authentication Protocols for OWA

And finally I get to the subject of this post, how to test if your Autodiscover configuration is running ok. I myself see 4 possibilities:

1. Microsoft Remote Connectivity Analyzer (recommended)

This tool will test the Autodiscover functionality and will provide you the returned XML information. Very complete. It also allows you to disable the SSL verification check which can be convenient if you are messing with non-commercial certificates. And it allows you to specify the user account in the Domain\Username format, which is convenient if your email domain differs from your AD UPN.

2. New Mail Profile using the control panel

Make sure Outlook is closed and then go the control panel.

You can easily add a new profile and configure an Exchange Account:

The great part of this wizard is that it’s able to prompt for additional credentials. You can only provide the e-mail address and the password. But TMG will not accept this. TMG is not able to find a user which matches this e-mail address as a user with this UPN in it’s domain. So to be more precise, the passwords entered over there don’t matter. You will be prompted anyway.

3. New Mail Profile from within Outlook

You’ll be presented with the same wizard of option 2. Disadvantage is that you’ll have to close Outlook before the mailbox can actually be accessed.

4. Test E-mail Autoconfiguration

You can launch the Test E-mail Autoconfiguration wizard by pressing ctrl and right-click the Outlook icon in the tray.

If you launch the wizard you’ll be presented with the following screen:

Now again you’re only asked for the e-mail address and the password. But unlike the configure new profile wizards presented above, this will NOT prompt for your account name in an other format. Somehow it seems to use the same sessions Outlook has established as an application.

So for instance:

1. Start outlook
2. Add new account from within outlook
3. Be prompted for credentials
4. Do NOT close outlook when prompted
5. Start the e-mail auto configuration wizard
6. E-mail auto configuration wizard succeeds

On the other hand:

1. Start outlook, don’t configure a profile or connect to an Exchange organization
2. Start the e-mail auto configuration wizard
3. E-mail auto configuration wizard will not succeed as there are no credentials to be-reused and no prompt appears!

In step 3 this is the error you will receive:

And on the log tab:

The error in words: Autodiscover to https://…/autodiscover/autodiscover.xml Failed (0x800C8203) or (0X80070057)

Test E-mail Auto Configuration Fact #1: it does not prompt for authentication

Besides that, suppose you got a connection using RPC over HTTPS to your mailbox. Try the E-mail Auto Configuration Test and provide the e-mail address of a colleague, fill in a dummy password. Yup, you do receive the information.

Test E-mail Auto Configuration Fact #2: you don’t have to provide the credentials of the mailbox you are querying for, as long as you are authenticated.

I just though I’d share this with the error code as it might lead you to think there are problems with your configuration whilst in fact everything is tip top.

Happy Discovering!

One of the problems with delegating permissions for a file system or Active Directory objects is the fact that the creator of the object is also the owner of an object.

Suppose you got someone who is a member of a group which grants him permission to create objects. This delegation would set the user who created the object as the owner of that object. Because he’s the owner he has full control regardless the delegation which is configured. Now suppose this person is removed from the group for one reason or another. In that case that person still has full control on the object he created because he is “owner”!...

This is where owner rights come in. You can restrict what permissions you get when you are the owner of the object. At my customers site this was configured to be just “read”. The owner rights principal is something from Windows 2008 and onwards. So when you are member of the group which got delegated permission you got: delegated to group permissions + owner right permissions = full control + read.
However once you are removed from the delegated group you have owner right permissions = read

Here’s some info from TechNet: http://technet.microsoft.com/en-us/library/dd125370(WS.10).aspx

How did we apply it in our environment?:

Added the “Owner Rights” entry with the following SACL on some top level OU’s: “Read (List Contents, Read all properties, Read permissions)” on “this object and all descendent objects”

So how does Exchange ActiveSync (EAS) comes into play? Well we seemed to have issues when user wanted to configure their device. They’d Always seem to end up with an error. On the CAS server we had the following error:

Log Name: Application

Source: MSExchange ActiveSync
EventID: 1053

Exchange ActiveSync doesn't have sufficient permissions to create the "<user object distinguished name>" container under Active Directory user "Active Directory operation failed on <Domain Controller FQDN>. This error is not retriable. Additional information: Access is denied. Active directory response: 00000005: SecErr: DSID-031521D0, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0 ". Make sure the user has inherited permission granted to domain\Exchange Servers to allow List, Create child, Delete child of object type "msExchangeActiveSyncDevices" and doesn't have any deny permissions that block such operations

I don’t have the time right now to go deeper into the troubleshooting process and the solution. In short this is what we noticed and what we did:

Whenever a user tries to configured an ActiveSync device, an object is created in Active Directory below the user object. More specifically a msExchActiveSyncDevices container object. And below this one an object is created for each device configured with ActiveSync.

• User [e.g. JohnDoe001]
• msExchActiveSyncDevices [container]
• msExchActiveSyncDevice [device#1]
• msExchActiveSyncDevice [Device#2]

The problem here is that the security for a newly created msExchActiveSyncDevices object is not correct. The Exchange schema prep should add the Exchange Servers with some permissions. In an environment without the Owner Rights configured everything works as the Exchange Servers are also the owners of these objects and thus have full control permissions.

The following procedure was performed so that we could leave the Owner Rights in place

1. Right-click the domain root and choose properties. Open the security tab:
2. Advanced => Add
3. Select the “Exchange Servers” principal
4. Check the permissions as shown in the screenshot:
• List contents
• Read all properties
• Write all properties
• Read Permissoins
• Modify Permissions
• Modify Owner

Make sure to select “Descendant msExchActiveSyncDevices” in the Apply to section.

A customer of mine was struggling with the following error in Outlook:

Microsoft Outlook: Cannot send this item

It started appearing after they introduced a new Exchange 2010 infrastructure and had migrated some pilot users from their Exchange 2003 environment. At first the repro we found was: start a new mail, type some random stuff in it and then wait. After some time (10 – 15’) hit send and see if you get the error. Now that was really lousy to reproduce…

On the internet we found a workaround: whenever the error would popup, you can choose the “Format Text” tab en switch between HTML and Rich Text. Then just hit send again.

We weren’t satisfied though, because of this formatting workaround we were suspecting a problem with the html of the signature in the mails. We found a hotfix regarding GIF images and outlook but that didn’t seemed to solve it. Besides that, our issue wasn’t always reproducible and time seemed to be a factor… So we shifted away from the formatting issues and looked further.

After some network tracing, lots of coffee and patience we seemed to have found the culprit. By clicking right on the outlook tray icon (while holding ctrl down) you can view some connection statistics:

And:

The picture is a bad example as I’m actually disconnected, but you get the idea. What we were seeing was that we had quit some failed requests. However this wasn’t really noticeable in the Outlook from the user point of view. Now we could reproduce our problem by opening several mails and just waiting until a failure popped up in the statistics. After such a failure we couldn’t send any of those mails we had open.

Once we learned this, we involved some networking guys to check all involved devices: firewalls, load balancers, … and they found out some device was malfunctioning (dropping packets).

Summary: if you are hitting the “outlook cannot send this item” error. Don’t go for the easy workaround by learning your users to change the format back and forth.  In stead do some troubleshooting and check the health of your network, at least if you are seeing connection failures in the statistics…

This evening I was installing an Exchange 2010 SP1 in a Lab environment which didn’t had Exchange before, and when going through the installation wizard I came across a checkbox I hadn’t seen before:

If I read the documentation correct, this could be an answer for those environments where Exchange management tasks are performed by an other team than the typical AD user management tasks. Here’s some TechNet info on the subject:

• Configure Exchange 2010 for Split Permissions
• Configure Exchange 2010 for Shared Permissions

This might come in handy whenever you are designing your delegation model for an Active Directory environment.

Using FIM it’s easy to provision mailboxes for users. In my current project we also needed to provision mail enabled users. The difference between a mailbox enabled and mail enabled is that the latter only has an email address associated with it’s account. The mailbox is typically located at an external organization.

Here’s a table from Exchange Provisioning using ILM 2007 and FIM 2010

It shows which attributes to flow for each recipient type. As we are going to provision mail enabled users we only need to flow

• mailNickname
• targetAddress

I like to use the sAMAccountName for the mailNickname. The reason why we wanted to mail enable some users is that we’d want external users (like consultants) to appear in the GAL. The customer itself was using a (fictive) domain like @internalAD.local.

At first I had my rules configured as:

• sAMAccountName –> mailNickname
• TargetAddress –> targetAddress

Where TargetAddress has values like “thomas@setspn.com”.

The result:

While it looks ok at first sight, when you look for the user in the GAL, it will appear with an e-mail address of the customer his address space...  Not exactly what we were looking for.

I did some testing using the Exchange Management Console, and the what I wanted to achieve seemed to be possible using the GUI, I must have done something wrong! It seems the targetAddress attribute in AD is supposed to contain “SMTP:emailaddress” and not merely “emailaddress”. So I changed the rules:

• sAMAccountName –> mailNickname
• TargetAddress –> “SMTP:”+targetAddress

And now my user looks like:

You can clearly see that the “primary” address is now the external address. And below you can see the difference in the GAL lookup. The first entry is bad, the second entry is OK.

In the project I’m currently involved, my colleagues are planning to migrate several mailboxes to a new Exchange 2010 platform. This particular issue was encountered when migrating mailboxes from Exchange 2007 (cross-forest) to Exchange 2010. They are currently running several test migrations so we can get an estimate of the amount of mailboxes we can migrate in a given timeframe. Using the New-MoveRequest cmdlet and some PowerShell magic they were processing 100 mailboxes one by one. Oddly a lot of them failed with the following error:

In words:

Failed to reconnect to Active Directory server dc.domain.tld. Make sure the server is available, and that you have used the correct credentials.
    + CategoryInfo          : NotSpecified: (0:Int32) [New-MoveRequest], RemoteTransientException
    + FullyQualifiedErrorId : A46C2901,Microsoft.Exchange.Management.RecipientTasks.NewMoveRequest

After trying the same New-MoveRequest command 5 or 6 times in a row all of a sudden the command finished successfully. Hmm that’s odd! As I’m sitting across my issue radar got activated and I launched google and stumbled upon the following topic: Not able to do a Cross forest Migration between E2k3-E2k7

The solution mentioned over there is to use the <FQDN of domain>\<username> format instead of the <NetBIOS domain name>\<username> when specifying the Remote Credentials. These credentials belong to a user which has permissions on the source mailboxes. Don’t ask for any logical explanation, but it sure did the trick.

For an overview of all supported logon credentials formats, see: KB929272: Interactive logon styles and Key Distribution Center account lookup in Windows Server 2003

In my current project I have an AD MA which is configured to provision for Exchange 2010. My customer has multiple Hub/CAS servers load balanced by a hardware solution. At first I provided a specific server in the Exchange 2010 RPS URI in the AD MA. Once the Exchange guys implemented the changes to Kerberos-enable the CAS array, I switched over to a virtual name. In our lab environment this worked just fine, however in the acceptance environment I kept receiving “ma-extension-error”:

This blocked all exports on the AD MA. In the  application event log the following error is show:

In text:

The most obvious thing to do at this time is to start googling. A very promising link is the Troubleshooting the Exchange Management Shell The above error could be caused by Release Candidate version PowerShell binaries. Well that’s definitely not the case. Using the following PowerShell commands I was able to reproduce the issue from the Synchronization Server:

• $user = Get-Credential
• $session = New-PSSession -configurationname microsoft.exchange -connectionuri http://virtualurl.demo.com/powershell -credential $user
• Enter-PSSession $session

Which gave me the following output:

Enter-PSSession : Starting a command on remote server failed with the following error message : The Windows Remote Shell cannot process the request; the selector value 962F626B-6492-493B-BF42-772C7C6BAFBD specified in the request was not found. For more information, see the about_Remote_Troubleshooting Help topic.

At line:1 char:16

+ Enter-PSSession <<<< $session
+ CategoryInfo : ResourceUnavailable: (:) [Enter-PSSession], PSRemotingTransportException
+ FullyQualifiedErrorId : System.Management.Automation.Remoting.PSRemotingDataStructureException,
Microsoft.PowerShell.Commands.EnterPSSessionCommand

So using netsh I started tracing and soon I saw the light, whereas we asked the networking guys to implement IP Affinity for the load balanced name, they seemed to have forgotten to actually do this… Hence the PowerShell session is being created with one server in the farm, whilst the actual command is probably being sent to another server… Not good!

Conclusion: Whenever you configure a load balancing solution for the Exchange 2010 CAS role, make sure you configure your solution to redirect sessions from the same IP to the same CAS node in the farm. This definitely seems to be a requirement for remote PowerShelling.

This week I attended “Designing and Planning AD Schema Extensions”, a session given by Brian Desmond at TEC Europe. During the session someone in the audience gave the remark that besides the “physicalDeliveryOffice” and “drink” attributes, the extensionAttribute1-15 are also often used to store company data. These extensionAttributes are in fact contributed by a schema extension from Exchange. They are there to use, but some Exchange actions do impact the content of these attributes!

Below are some screenshots of the timestamps which show which attributes were touched after doing a certain action, In the first one I created a new user and using ADUC I’ve set a value for each extensionAttribute. The user was created at 15:52, the attributes were set at 15:55.

After using the Exchange 2010 Management Tools  to create a mailbox for this user, the timestamps look like the screenshot below, you can see a lot of attributes got added at 15:58, but the extensionAttributes didn’t got touched. So no problems so far.

After Disabling the mailbox using the Exchange 2010 Management Tools things look differently, besides the expected Exchange related attributes, also all of the extensionAttributes are touched at the exact same time: 16:01. In fact, using ADUC you can verify that all extensionAttributes are empty.

This doesn’t have to be a problem, as long as you take it into account. Perhaps if you have FIM in your environment, things get automagically corrected afterwards. Perhaps emptying the attributes is part of your deprovisioning process anyhow. But on the other hand, this might also be the explanation why some users have lost the content for these attributes. Besides the impact of Exchange, using the extensionAttributes however is tricky, you never know what third party application is going to store its data there. As Brian said during his session, don’t be afraid to extend the schema, just think/plan it thoroughly.

There are several good resources on how to provision mailboxes for Exchange 2010 using FIM 2010. For starters the following attributes have to be set (export attribute flow) during the user creation phase:

• Mailnickname
• homeMDB
• msExchHomeServerName

Secondly the AD MA should be configured to "provision for Exchange 2010" and the RSP URI should be provided as well. The URL should looke like http(s)://casarray.yourdomain.tld/Powershell.

Following these steps FIM will create a mailbox-enabled account when provisioning accounts in Active Directory. An issue which I came upon is the options panel in Exchange 2010 Outlook Web Access (OWA). All my FIM provisioned users can successfully logon to OWA, but when clicking the options button they received the following error:

I verified the application event log of the Exchange CAS server around the time I did my tests and came upon the following error:

A copy paste of the error:

(Process w3wp.exe, PID 3104) "RBAC authorization returns Access Denied for user test001@corp.contoso.com. Reason: No role assignments associated with the specified user were found on Domain Controller CORP-DC-01.corp.contoso.com"

Using the Exchange Management Shell I was able to verify the RoleAssignmentPolicy of the users. I could clearly see that all users created using the Exchange Management Console had the "Default Role Assignment Policy" assigned and that the users provisioned from within FIM had no policy assigned. Verifying the RoleAssignmentPolicy can be done as follows:

• for a specific user: get-mailbox accountname | ft name,roleassignmentpolicy
• for all users: get-mailbox | ft name,roleassignmentpolicy

Some examples:

Using the update-recipient cmdlet I could actually reproduce this behavior without FIM in the picture. It looks like the new-mailbox cmdlet sets the default policy, but the update-recipient cmdlet which can convert a legacy mailbox to a mailbox does not set this policy. Guess what FIM is using…

If you'd want to fix this, you could issue the following command:

• get-mailbox | set-mailbox –RoleAssignmentPolicy "default role assignment policy"

A colleague of me pointed me out to the msExchRBACPolicyLink attribute of a mailbox-enabled user account in AD. This attributes holds the RoleAssignmentPolicy and looks like CN=Default Role Assignment Policy,CN=Policies,CN=RBAC,CN=MAIL,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=CORP,DC=CONTOSO,DC=COM. When I add this attribute to the list of attributes I initially mentioned all goes well. Mailbox created, RoleAssignmentPolicy in place and Exchange Control Panel (options) available!

A possibility to find out which setting is managed by which attribute in AD is to leverage the repadmin command with the showobjmeta option:

• repadmin /showobjmeta name_of_dc dn_of_userobject

The name_of_dc can be one specific domain controller, or you can use a wildcard (*) in your name to target multiple DC’s. Following the DC you can provide the DN of your test user. Now just change a setting (like in this case the RoleAssignmentPolicy) using PowerShell, and then execute the repadmin command. That way you can find out which attribute was changed seconds ago and hence is very likely to represent the setting you can control with FIM. As a (not related) example I got the output of a useraccount which I just disabled in AD. You can clearly see that this is controlled by the “userAccountControl” attribute:

So to conclude I think as a bare minimum the following attributes should be exported when provisioning mailboxes for Exchange 2010:

[Update 21/12/2010] This topic was discussed on the FIM TechNet Forums as there’s some official Technet Documentation out there now, I would like to reference here. For a fully supported Kerberos enabled CAS array, Exchange 2010 SP1 is required, and you can follow this guide: http://technet.microsoft.com/en-us/library/ff808313.aspx

One of the problems I had with FIM 2010 in the past is the configuration of the RPS URI in the FIM Active Directory Management Agent (AD MA) configuration. When you want FIM to provision mailbox’ for you, or when using FIM for GALsync (provision contacts) you have to provide the URL to the exchange PowerShell webservice (http://webmail.contoso.com/powershell):

A lot of Exchange environments have more than one CAS server, typically these are combined with a NLB solution. So ideally you just point FIM to the CAS URL (like webmail.contoso.com) and thus ensuring FIM will not be dependent on the uptime of one specific CAS server. Because the MA only uses Kerberos to connect to that URL, there is a problem. Out of the box the CAS array URL is not kerberos enabled. As such the provisioning of mailbox’ will fail. When targeting a specific CAS server Kerberos authentication goes just fine. Because all required SPN’s are registered for each CAS server by the exchange setup and thus Kerberos authentication against a CAS server works.

From Exchange 2010 RU3 (Or RU4, not quit sure) there is a way to enable the CAS array for Kerberos. The word is out that this would officially be supported from Exchange 2010 SP1. The following steps are required:

1. Create a service account in Active Directory: svc_exchange
2. Register the SPN’s on this account (I always register both FQDN and short):
   • setspn -F -S exchangeMDB/webmail.contoso.com CONTOSO\svc_exchange
   • setspn -F -S exchangeRFR/webmail.contoso.com CONTOSO\svc_exchange
   • setspn -F -S exchangeAB/webmail.contoso.com CONTOSO\svc_exchange
   • setspn -F -S HTTP/webmail.contoso.com CONTOSO\svc_exchange
   • setspn -F -S exchangeMDB/webmail CONTOSO\svc_exchange
   • setspn -F -S exchangeRFR/webmail CONTOSO\svc_exchange
   • setspn -F -S exchangeAB/webmail CONTOSO\svc_exchange
   • setspn -F -S HTTP/webmail CONTOSO\svc_exchange
3. The following steps have to be executed on each CAS server
   • $cred = get-credential "CONTOSO\svc_exchange"
   • Set-clientaccessserver -Identity CASserver -AlternateServiceAccountCredential $cred
   • iisreset
   • restart-service MSexchangeAB
   • restart-service MSexchangeRPC
4. To verify the setting, execute the following on each CAS server
   • Get-ClientAccessServer -Identity CASserver -IncludeAlternateServiceAccountCredentialStatus |fl AlternateServiceAccountConfiguration

From now on, if you use outlook to connect to the CAS array, or just when provisiong with FIM and using HTTP, Kerberos will be used. You can eassily verify this with Klist (Windows 2008 and up built-in) or with Kerbtray. Or just in the security eventlog of a CAS server.

The actual procedure I used comes from the following blog:  TUTORIEL: Exchange 2010 SP1 (Beta) - Activer Keberos pour un pool de serveurs CAS (Client Access Array) As it’s French, I think it’s justified to have some English alternative.

Technet help for the set-clientaccessserver command: http://technet.microsoft.com/en-us/library/bb125157.aspx