Trials of a Network Admin

If you’re going to be using Office 365 in any capacity, you essentially need to setup Active Directory Federation Services (ADFS). This will sync your on-premise AD to Microsoft’s cloud and allow your users to sign-on to Office 365 using their domain credentials. I did most of this setup using ADFS 2.0 on Windows Server 2008 R2 about a year ago when we first starting using some O365 features and thought everything was working fine. Of course if it was, this post wouldn’t exist.

For the 2008 R2 variant of ADFS (there’s a newer version on Server 2012), Microsoft recommends five (5!) servers: one to sync your AD to the cloud, two load-balanced as the actual ADFS servers to authenticate users, and two more (also load-balanced) as ADFS proxies, which would typically sit in your DMZ and relay authentication requests to your ADFS server(s).

NOTE: A Microsoft Office 365 expert I saw speak at a recent presentation indicated that installing the DirSync tool on a domain controller is now supported, so that would at least eliminate one of the five servers mentioned above. I haven’t verified that elsewhere, and I’m not sure if it’s only applicable to ADFS on Server 2012, but it’s worth looking into if you’re doing a setup from scratch.

I balked at installing five servers, especially as we were (and for the most part, still are) in a testing phase with Office 365. I setup the AD sync server and one ADFS server and started testing. After the usual teething issues, everything looked good and users could authenticate to Office 365 using single-sign-on (SSO) from behind the corporate firewall. That left dealing with users outside the firewall. Basically you can handle this one of two ways; either install the ADFS proxy server(s) as per Microsoft, or expose port 443 on the ADFS server to the Internet. The later choice is considered slightly riskier from a security perspective, but unless you’re going to put the proxy server(s) in a DMZ, there’s really no (security) difference that I can see. I went with the easier approach, which as it turns out was a mistake.

After opening port 443, I tested ADFS from a non-domain joined PC outside our firewall. As expected, I was prompted for credentials and was able to login successfully. I also tested using my domain-joined laptop (running Win 7) from home, and was likewise able to login. So everything was good as far as we knew.

Until we started testing Windows 8.1, that is. I noticed pretty quickly that the domain-joined 8.1 machines could not connect to O365 when running off-domain. So for example, a user with a domain-joined tablet could not connect when using that tablet from home, yet back in the office everything worked fine. A non-domain joined Windows 8/8.1 system would however prompt for credentials and allow the user to login.

At this point I re-tested this same scenario on Windows 7 and found that most of the time, the Win 7 client could connect from outside the domain. Sometimes though I would repeatedly be prompted for credentials and could not connect. Inconsistent behavior. Lovely.

I opened a ticket with Microsoft, who, to their credit, provides free support for O365 on top of providing O365 itself for free to K-12 schools. It took a while, but the ADFS team I was working with eventually established that ADFS was working properly and that Kerberos authentication was failing. Of course troubleshooting that would require involving the directory services team, so I’d need to open a new ticket with them. Sigh.

Before I go any further, let me summarize the behavior I was seeing:

Incidentally, a login failure showed in the browser as “page could not be displayed,” but was actually a “401 Unauthorized” reply from the ADFS server.

Having now been pointed in the direction of Kerberos, I decided to do some more troubleshooting before opening a ticket with the directory services team. I stumbled on this KB article, which came pretty close to describing the problem we were experiencing. Unfortunately I went through that entire article and none of the resolutions shown had any effect.

That KB article did at least force me to look at web.config and the different types of authentication that ADFS supported.

I knew “Integrated” authentication was working fine behind the firewall, using Kerberos. And likewise I knew “forms”-based authentication worked from outside the firewall. Now, in theory, the client should be trying these authentication methods in order, negotiating the most secure one available to both the server and the client. And that seems to be the cause of the problem.

Now, I’ll freely admit that Kerberos and negotiating authentication methods is not one of my areas of expertise, but a packet-trace showed a clear problem. The 8.1 client was attempting to contact a domain controller from outside the firewall, to find a Kerberos SRV record; obviously there’s no way that would work. So my best guess is this: when joined to the domain, the client believes it can/should be able to authenticate with Kerberos; this prevents the negotiation of the authentication method from “falling-back” to forms-based. But without connectivity to a domain controller, Kerberos authentication fails, at least most of the time. I can’t explain the inconsistent behavior on the Win 7 systems, other than guessing that I was switching a test machine from my domain network to my off-domain network so quickly that the client didn’t need to request another Kerberos ticket.

So after all this, I ended up coming full circle to the ADFS proxy server I chose not to install initially. Apparently while you don’t need an ADFS proxy for most functionality, getting things working 100% does require one. After installing the proxy and making the necessary firewall changes, all of our problems went away. (See this link for a quick rundown on installing and configuring an ADFS proxy. Keep in mind there is a newer update rollup than the article references.)

The reason for this is simple: the ADFS proxy is only setup for forms-based authentication. Using split-brain DNS, an internal client connects to your ADFS server and authenticates with Kerberos, but an external client connects to the ADFS proxy and is (always) prompted for credentials via forms-based authentication.

In retrospect I should have just followed Microsoft’s recommendations and installed the proxy from the beginning. Unlike the ADFS setup itself – which is quite time consuming – installing and configuring the proxy is quick and easy. It’s just another server to manage. But then again, what would modern IT be without virtual server sprawl?

With that, another obscure problem comes to a resolution. If this helps you please leave a comment; I love getting feedback.

Bonus Info: If you want to customize the ADFS proxy login page to make it look better, check out this page.

I also recommend making the changes shown at the bottom of this page to FormsSignIn.aspx.cs so that the username field automatically gets populated.

Finally, after making the changes above, you can make one more change to FormsSignIn.aspx.cs so that the focus is set to the password field. Find UsernameTextBox.Text = userName; in FormsSignIn.aspx.cs and add PasswordTextBox.Focus(); right below it.

With those changes made, you’ll have a much better looking login page that will automatically populate the username field and set the focus to the password field. Not bad for a few minutes work.