802.1x Certificate Based Authentication against NPS on Windows Server 2016

Posted by James F. Prudente on September 20, 2018
Posted in: PKI, Windows Server. Tagged: , , . Leave a comment

When we were preparing to upgrade our domain controllers from 2008 R2 to 2016, we of course began inventorying all functions that were going to be migrated. One of these was Network Policy Server (NPS). NPS is one of the easiest services to migrate to a new system, since it’s basically a simple backup and restore, and we weren’t expecting any difficulties. Of course had things gone smoothly, this post wouldn’t exist.

A little background:

We use NPS for multiple functions, including Cisco AAA and wireless device authentication. Our Meraki wireless infrastructure has multiple SSIDs, some of which are for domain-joined devices, and others for non-domain devices. The domain-joined devices use 802.1x certificate-based authentication and the non-domain joined devices use various other methods.

NPS was migrated from 2008 R2 to 2016 and everything other than the 802.1x certificate authentication worked. The 2016 servers were correctly setup as RAS servers and had certificates that were valid for client & server authentication, as needed. I spent quite a long time troubleshooting this with no success, and eventually had to back-burner it. The 2008 R2 servers were due to be retired, so as a temporary measure I migrated NPS to a couple 2012 R2 servers, all of which worked flawlessly after about two minutes worth of effort.

This was pretty early in 2016’s lifecycle, so since it worked in 2008 R2 and 2012 R2, we chalked it up to a bug in the OS and intended to pursue it with Microsoft at a later date.

Before I had a chance to call Microsoft, another project had us looking at our PKI configuration. We were having issues using LDAPS with this particular system (despite using it elsewhere), and after extensive troubleshooting the vendor told us they thought the hash algorithm used by our PKI was unsupported.

Our PKI was originally setup on a 2003 server back when that was the current server OS. It had been migrated to a newer server since then but the hash algorithm had never been changed; it was still using md5RSA. Now upgrading the hash algorithm isn’t something to be taken lightly, as it can break a lot of things, but in our case since our PKI is pretty simple I decided (after backing everything up) that the upgrade risk was minimal. I changed the hash algorithm to SHA256, reissued RAS certificates to the NPS servers and just like that things started working.

I can’t find any changelog indicating why NPS would not work with an older certificate hash on 2016 but empirical evidence shows the hash was the problem.

There are plenty of resources about upgrading your root CA’s hash algorithm, but this is a good start from Jason Bender at Microsoft.

Managing Mail-Enabled Security Groups in O365

Posted by James F. Prudente on September 7, 2018
Posted in: Active Directory, Exchange, Office 365, PowerShell. 3 Comments

Office 365 certainly has its plusses and minuses, and one of the areas that clearly falls in the latter category is how O365 handles mail-enabled security groups.

Consider the following scenario:

  • You have an on-premise Active Directory sync’d to O365/Azure via ADSync.
  • You have an on-premise security group that you mail-enable, creating an e-mail address @yourdomain.com.
  • This mail-enabled group syncs to O365, and then exists in Exchange Online.
  • You then mail-disable the group on-premise and wait for the change to sync to Exchange Online.

Logic would dictate that once the mail-enabled attribute was removed from the group, it would disappear from Exchange Online. Of course it’s not that simple.

What you will instead find is that the security group still shows in Exchange Online, however it now has a proxy address @yourdomain.onmicrosoft.com. It will also continue to be visible in both the online and offline address books. If you are trying to move groups to the cloud and/or cleanup your distribution lists, this can cause confusion.

The fix fortunately is easy, but it’s an extra step that shouldn’t be necessary IMO. Credit to Tim McMichael’s TechNet post for the details.

The process to properly remove a mail-enabled security group from Exchange Online is as follows:

  • On-Premise, remove the mail-enabled attribute from the group using the Disable-DistributionGroup PowerShell command.
  • On Exchange Online, find the group via Get-MsolGroup –SearchString “Group Name”.
  • Once you’ve confirmed you have the right group, remove it via Get-MsolGroup –SearchString “Group Name” | Remove-MsolGroup

This will finally remove it from Exchange Online and your address books.

EDIT: There is a major caveat here to be aware of. If you are using mail-enabled security groups to grant permissions to shared calendars, you CANNOT remove the mail attributes or you will break access permissions. In that situation I would advise simply hiding the list from the address book.

Monitoring Microsoft NLB with PowerShell

Posted by James F. Prudente on June 21, 2018
Posted in: Active Directory, ADFS, PowerShell, Windows Server. Tagged: , , , . Leave a comment

Recently the need arose to monitor the status of a Microsoft NLB cluster. There are a number of ways of approaching this, but PowerShell seemed like the cleanest.

I found this script on TechNet but found that it did not work in our (2008 R2) environment. During troubleshooting, what I found was that instead of returning a single status result, the WMI query was returning an array of results for each server. It appears that the statuscode is only relevant for the specific hostpriority applicable to each server.

For instance, this is the server with hostpriority 1 as shown in the NLB Manager:

And this is the one with hostpriority 2:

As you can see, a proper statuscode is only returned for the current hostpriority for each server.

With that in mind I made a change to the script to enumerate through the array of results for each server and check the status accordingly. This has only been tested on ADFS 2.0 on 2008 R2 servers, and it would need to be revised if you have more than two servers in a NLB cluster.


#Define Nodes
$node1 = "SERVER01"
$node2 = "SERVER02"

#get NLB status on NLB Nodes
$Node1status = Get-WmiObject -Class MicrosoftNLB_Node -computername $node1 -namespace root\MicrosoftNLB | Select-Object __Server, statuscode, status, hostpriority
$Node2status = Get-WmiObject -Class MicrosoftNLB_Node -computername $node2 -namespace root\MicrosoftNLB | Select-Object __Server, statuscode, status, hostpriority

$node1converged = $false

Foreach ($status in $Node1status)
{
if(($status.hostpriority -eq "1" -and $status.statuscode -eq "1008") -or ($status.hostpriority -eq "2" -and $status.statuscode -eq "1007"))
{
$node1converged = $true
write-host "NLB Status of $node1 is: Converged"
}
}

Foreach ($status in $Node2status)
{
if(($status.hostpriority -eq "1" -and $status.statuscode -eq "1008") -or ($status.hostpriority -eq "2" -and $status.statuscode -eq "1007"))
{
$node2converged = $true
write-host "NLB Status of $node2 is: Converged"
}
}

Sorry about the indents; they show up in the WordPress editor but not on the actual page.

Let me know if you find this useful, or if you are able to use it on a platform other than 2008 R2.

IIS Application Pool Will Not Start – App Log Error 2307

Posted by James F. Prudente on June 13, 2018
Posted in: Windows Server. 1 Comment

After applying patches to a 2012 R2 server, I found one of our IIS websites was not working. While IIS was itself running, the DefaultAppPool was stopped. Attempts to start the app pool were unsuccessful; the pool would stop almost immediately, and error 2307 from “IIS-W3SVC-WP” was logged in the Application Log.

The error message itself contained the following information:

The worker process for application pool 'DefaultAppPool' encountered an error 'Cannot read configuration file' trying to read configuration data from file '\\?\<EMPTY>', line number '0'. The data field contains the error code.

Not the most helpful error message since it doesn’t specific the file that cannot be read. Some research online turned up references to specific IIS config files (web.config, etc.) having incorrect user permissions, but I looked into that and everything seemed correct. I used Sysinternals Process Monitor to see if I could determine the file in question, with no luck.

Eventually I stumbled on a post (which I unfortunately lost track of), which referenced a Microsoft KB article. The article pertains to Windows 10 and Server 2016, the 1709 Fall Creators Update, and a totally different event ID from a totally different service. That said, the resolution shown fixed the error I was having.

The actual fix is quick:

net stop WAS /y
rmdir /s /q C:\inetpub\temp\appPools
net start W3SVC

Evidently the patches I applied caused the same issue (broken symbolic links) as the Win 10 1709 upgrade.

Inaccessible Network Shares on Windows 10 1709

Posted by James F. Prudente on November 15, 2017
Posted in: Permissions, Windows 10, Windows Server. 13 Comments

After upgrading a number of systems to the Fall Creators edition of Windows 10, version 1709, we began receiving reports of certain users being unable to access a specific set of shared folders. These were folders hosted on a 2012R2 file server, and the error received indicated the share was missing, rather than there being a permissions issue.

Searching for this type of error on the Fall Creators edition turns up two common results:

We were pretty certain SMBv1 was not in use but ran Wireshark to sniff the SMB traffic and verify SMB2 was being used.

Likewise, we knew guest account access was already disabled, and the Microsoft KB article references log entries that would be present if this were causing a problem. There were no relevant log entries, so we crossed this off the list of possible causes.

Interestingly, not all shares were an issue; even other ones on the same server worked fine. That said, this share had been used for years without problems, with multiple versions of Windows, including three previous Win 10 builds.

I started reviewing the permissions on the share, just to see what might be different from other shares that were working, and quickly found a discrepancy.

The share in question was the parent folder for many sub-folders. The sub-folders have different permissions for different groups. We provide a shortcut directly to the parent folder and have access-based enumeration enabled so that users only see the folders they have access to. Nothing all that unusual here.

The parent folder had “Authenticated Users” being granted “Traverse Folder / Execute File” on the folder itself. (“This Folder Only”)

Certain user groups however had “Read” permissions on the parent folder. Some quick testing showed that the users with “Read” permissions could access the share, while those with “Traverse Folder” could not.

The entire purpose of “Traverse Folder” is to “[a]llow..moving through a restricted folder to reach files and folders beneath the restricted folder in the folder hierarchy.” Having this permission should be perfectly sufficient for what we were doing, and it has been for years, up until the Fall Creators version of Win 10.

Even stranger though is that “bypass traverse checking,” which is granted to everyone by default (and enabled in our domain) is supposed to entirely negate the need for the “Traverse Folder” right in the first place.

Something obviously changed in Fall Creators that has broken traverse checking; my guess is with all the file system work that Microsoft acknowledges they put into Fall Creators, they either inadvertently broke this or deliberately changed the behavior without communicating it to us. Although if this were a deliberate change it completely eliminates the value of traverse bypass in the first place.

The fix here was easy; I gave “Read” permissions to “Authenticated Users” for the parent folder only. This had no effect on any of the sub folders or users access to them, but it resolved the path not found error.

Oh well…back to beta testing the latest GA build of Win 10…