Managing Mail-Enabled Security Groups in O365

Posted by James F. Prudente on September 7, 2018
Posted in: Active Directory, Exchange, Office 365, PowerShell. 3 Comments

Office 365 certainly has its plusses and minuses, and one of the areas that clearly falls in the latter category is how O365 handles mail-enabled security groups.

Consider the following scenario:

  • You have an on-premise Active Directory sync’d to O365/Azure via ADSync.
  • You have an on-premise security group that you mail-enable, creating an e-mail address @yourdomain.com.
  • This mail-enabled group syncs to O365, and then exists in Exchange Online.
  • You then mail-disable the group on-premise and wait for the change to sync to Exchange Online.

Logic would dictate that once the mail-enabled attribute was removed from the group, it would disappear from Exchange Online. Of course it’s not that simple.

What you will instead find is that the security group still shows in Exchange Online, however it now has a proxy address @yourdomain.onmicrosoft.com. It will also continue to be visible in both the online and offline address books. If you are trying to move groups to the cloud and/or cleanup your distribution lists, this can cause confusion.

The fix fortunately is easy, but it’s an extra step that shouldn’t be necessary IMO. Credit to Tim McMichael’s TechNet post for the details.

The process to properly remove a mail-enabled security group from Exchange Online is as follows:

  • On-Premise, remove the mail-enabled attribute from the group using the Disable-DistributionGroup PowerShell command.
  • On Exchange Online, find the group via Get-MsolGroup –SearchString “Group Name”.
  • Once you’ve confirmed you have the right group, remove it via Get-MsolGroup –SearchString “Group Name” | Remove-MsolGroup

This will finally remove it from Exchange Online and your address books.

EDIT: There is a major caveat here to be aware of. If you are using mail-enabled security groups to grant permissions to shared calendars, you CANNOT remove the mail attributes or you will break access permissions. In that situation I would advise simply hiding the list from the address book.

Monitoring Microsoft NLB with PowerShell

Posted by James F. Prudente on June 21, 2018
Posted in: Active Directory, ADFS, PowerShell, Windows Server. Tagged: , , , . Leave a comment

Recently the need arose to monitor the status of a Microsoft NLB cluster. There are a number of ways of approaching this, but PowerShell seemed like the cleanest.

I found this script on TechNet but found that it did not work in our (2008 R2) environment. During troubleshooting, what I found was that instead of returning a single status result, the WMI query was returning an array of results for each server. It appears that the statuscode is only relevant for the specific hostpriority applicable to each server.

For instance, this is the server with hostpriority 1 as shown in the NLB Manager:

And this is the one with hostpriority 2:

As you can see, a proper statuscode is only returned for the current hostpriority for each server.

With that in mind I made a change to the script to enumerate through the array of results for each server and check the status accordingly. This has only been tested on ADFS 2.0 on 2008 R2 servers, and it would need to be revised if you have more than two servers in a NLB cluster.


#Define Nodes
$node1 = "SERVER01"
$node2 = "SERVER02"

#get NLB status on NLB Nodes
$Node1status = Get-WmiObject -Class MicrosoftNLB_Node -computername $node1 -namespace root\MicrosoftNLB | Select-Object __Server, statuscode, status, hostpriority
$Node2status = Get-WmiObject -Class MicrosoftNLB_Node -computername $node2 -namespace root\MicrosoftNLB | Select-Object __Server, statuscode, status, hostpriority

$node1converged = $false

Foreach ($status in $Node1status)
{
if(($status.hostpriority -eq "1" -and $status.statuscode -eq "1008") -or ($status.hostpriority -eq "2" -and $status.statuscode -eq "1007"))
{
$node1converged = $true
write-host "NLB Status of $node1 is: Converged"
}
}

Foreach ($status in $Node2status)
{
if(($status.hostpriority -eq "1" -and $status.statuscode -eq "1008") -or ($status.hostpriority -eq "2" -and $status.statuscode -eq "1007"))
{
$node2converged = $true
write-host "NLB Status of $node2 is: Converged"
}
}

Sorry about the indents; they show up in the WordPress editor but not on the actual page.

Let me know if you find this useful, or if you are able to use it on a platform other than 2008 R2.

IIS Application Pool Will Not Start – App Log Error 2307

Posted by James F. Prudente on June 13, 2018
Posted in: Windows Server. 1 Comment

After applying patches to a 2012 R2 server, I found one of our IIS websites was not working. While IIS was itself running, the DefaultAppPool was stopped. Attempts to start the app pool were unsuccessful; the pool would stop almost immediately, and error 2307 from “IIS-W3SVC-WP” was logged in the Application Log.

The error message itself contained the following information:

The worker process for application pool 'DefaultAppPool' encountered an error 'Cannot read configuration file' trying to read configuration data from file '\\?\<EMPTY>', line number '0'. The data field contains the error code.

Not the most helpful error message since it doesn’t specific the file that cannot be read. Some research online turned up references to specific IIS config files (web.config, etc.) having incorrect user permissions, but I looked into that and everything seemed correct. I used Sysinternals Process Monitor to see if I could determine the file in question, with no luck.

Eventually I stumbled on a post (which I unfortunately lost track of), which referenced a Microsoft KB article. The article pertains to Windows 10 and Server 2016, the 1709 Fall Creators Update, and a totally different event ID from a totally different service. That said, the resolution shown fixed the error I was having.

The actual fix is quick:

net stop WAS /y
rmdir /s /q C:\inetpub\temp\appPools
net start W3SVC

Evidently the patches I applied caused the same issue (broken symbolic links) as the Win 10 1709 upgrade.

Inaccessible Network Shares on Windows 10 1709

Posted by James F. Prudente on November 15, 2017
Posted in: Permissions, Windows 10, Windows Server. 13 Comments

After upgrading a number of systems to the Fall Creators edition of Windows 10, version 1709, we began receiving reports of certain users being unable to access a specific set of shared folders. These were folders hosted on a 2012R2 file server, and the error received indicated the share was missing, rather than there being a permissions issue.

Searching for this type of error on the Fall Creators edition turns up two common results:

We were pretty certain SMBv1 was not in use but ran Wireshark to sniff the SMB traffic and verify SMB2 was being used.

Likewise, we knew guest account access was already disabled, and the Microsoft KB article references log entries that would be present if this were causing a problem. There were no relevant log entries, so we crossed this off the list of possible causes.

Interestingly, not all shares were an issue; even other ones on the same server worked fine. That said, this share had been used for years without problems, with multiple versions of Windows, including three previous Win 10 builds.

I started reviewing the permissions on the share, just to see what might be different from other shares that were working, and quickly found a discrepancy.

The share in question was the parent folder for many sub-folders. The sub-folders have different permissions for different groups. We provide a shortcut directly to the parent folder and have access-based enumeration enabled so that users only see the folders they have access to. Nothing all that unusual here.

The parent folder had “Authenticated Users” being granted “Traverse Folder / Execute File” on the folder itself. (“This Folder Only”)

Certain user groups however had “Read” permissions on the parent folder. Some quick testing showed that the users with “Read” permissions could access the share, while those with “Traverse Folder” could not.

The entire purpose of “Traverse Folder” is to “[a]llow..moving through a restricted folder to reach files and folders beneath the restricted folder in the folder hierarchy.” Having this permission should be perfectly sufficient for what we were doing, and it has been for years, up until the Fall Creators version of Win 10.

Even stranger though is that “bypass traverse checking,” which is granted to everyone by default (and enabled in our domain) is supposed to entirely negate the need for the “Traverse Folder” right in the first place.

Something obviously changed in Fall Creators that has broken traverse checking; my guess is with all the file system work that Microsoft acknowledges they put into Fall Creators, they either inadvertently broke this or deliberately changed the behavior without communicating it to us. Although if this were a deliberate change it completely eliminates the value of traverse bypass in the first place.

The fix here was easy; I gave “Read” permissions to “Authenticated Users” for the parent folder only. This had no effect on any of the sub folders or users access to them, but it resolved the path not found error.

Oh well…back to beta testing the latest GA build of Win 10…

WSUS Error 0x80092026 – File Failed Post-Processing

Posted by James F. Prudente on October 16, 2017
Posted in: Windows 10. Leave a comment

As is typically the case with my sporadic blog posts, we recently ran into a situation where, as a colleague likes to put it, we ended up far down the rabbit hole. There are a few lessons to be learned here so I’m going to explain a bit about the troubleshooting process, not just how we eventually fixed the issue.

The initial complaint as reported was that one set of laptops was not working properly, but only during the first class period, and only in one classroom. Now, it’s not uncommon for mobile devices to take a little while to wake up, but in this case the machines were turning on an hour before class started, and again, we were told the problem was very limited in scope.

We were able to duplicate the problem, but again, only during the one period, and even stranger, only when the class was in the room. We use Meraki access points, and based on this particular AP not seeing its mesh neighbors all that well, we initially thought we were dealing with some localized RF interference. After extensive troubleshooting of the RF environment we could not find any explanation for the apparent interference. The AP was seemingly overloaded during this period of time, but there was no obvious cause.

One thing that did stand out was certain wireless clients were transferring an abnormal amount of data. We noted this with the intent to revisit it later, but because we had no reports of issues from other locations, we concluded this was unrelated to our current situation.

That was a mistake. We assumed that because there were no other problems reported, that there were no other problems. As it turns out, nobody else was using laptops at this time of day. The lesson here is to check your assumptions.

Once we realized that we could not draw any conclusions from the location of the problem, we expanded our investigation and revisited the previously mentioned excessively large amounts of data being downloaded. The Meraki APs do an excellent job of reporting certain info, and it quickly became obvious the data was coming from our WSUS patch server.

We had recently rebuilt WSUS, and with 2012 R2 there are (were) some important out-of-band patches that can result in failed downloads. We reviewed these and confirmed that on a fully-patched 2012 R2 install, these needed updates are now included in the normal security and critical updates that Microsoft pushes out. (Evidently you still must add the .esd mime type to IIS, but that’s unrelated to this post.)

We next turned to one of the laptops that was experiencing a problem. On Windows 10, getting a human-readable update log requires running the PowerShell command Get-WindowsUpdateLog; this creates the old-style WindowsUpdate.log you may be used to from earlier Windows versions.

Searching for errors, we came across the following:

Misc Validating signature for <file guid> with dwProvFlags 0x00000080:
Misc Error: 0x80092026 when verifying trust for <file guid>
DownloadManager File failed postprocessing, error = 80092026
DownloadManager Failed file: URL = <WSUSServer><File>, Local path = <local path>
DownloadManager Error 0x80092026 occurred while downloading update; notifying dependent calls.

I should add that many of the affected laptops were not consistently reporting to the new WSUS server, we were not certain if this was related but it certainly seemed suspicious.

A number of online searches confirmed that this error had something to do with a certificate check failing for a particular update. Since our content filter does SSL decryption, we double-checked that we were not decrypting any traffic to Microsoft’s CRL or patch servers: we were not. Of course, if something were being blocked by the content filter we would expect every client to be impacted, but most of our PCs were working fine.

We found different files failing post-processing on different computers, so this did not seem to be related to a specific update. Was what more interesting is that despite only one downloaded update failing, WSUS (or BITS) seemingly discarded the entire batch of downloads, so the next time Windows Update started to run, it re-downloaded everything.

After further troubleshooting, I stumbled on this thread on Microsoft TechNet. User “lforbes” posted the following:

I found the offending key for those that are interested.
HKEY_USERS\S-1-5-18\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing
State SHOULD be 146432 or 0x00023c00
If it switches to 408576 or 0x00063c00 [t]hen it will cause these Crypto Errors

Checking the registry on affected computers confirmed that the key in question was set to 0x00063c00. After changing it back to 0x0023c00, the trust provider errors stopped. I have no idea how he found that key, and what’s amazing is that this TechNet post seems to be the only reference to this error and this registry key out there.

I didn’t want to just make the change without understanding what it meant, so researching further lead to this post which clarifies that the “State” key is a bitwise flag for various settings of “Microsoft Trust verification services, which provide a common API for determining whether a specific subject can be trusted.” After a bit of hex to binary conversion and a comparison between the two sets of flags, I determined that the difference was the incorrect setting had enabled the flag to only “allow…items in personal trust database.” Seemingly this flag was preventing a system-level certificate from being used to validate a Microsoft signature. No clue why.

But of course things didn’t stop there.

Whether it was related to the original certificate error or not, many of the same PCs that had this error were also not reporting to WSUS. This at least I already had a partial fix for. Years ago I had put together a basic batch file that reset the SusClientId, deleted the SoftwareDistribution folder, and basically forced a full reset of the WSUS client. (I’m not taking credit for figuring this all out, but I have no idea where I first saw the info.)

The process involves running the following as a batch file on the affected PC:

net stop wuauserv

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v AccountDomainSid /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v PingID /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientId /f

reg delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientIdValidation /f

del c:\windows\softwaredistribution /S /F /Q

net start wuauserv

wuauclt /resetauthorization /detectnow

This stops the Automatic Update service, clears out the registry keys that identity the system to WSUS, clears out the patch download cache, restarts the Automatic Update service, and then re-registers the system to WSUS. There’s one little problem. The “/detectnow” switch was apparently removed in the Win 10 version of wuauclt.exe, leaving no simple way to force a machine to scan WSUS for updates.

There are a couple of ways to get around this via VBScript or languages that directly call the Windows Update API. In our case however we already were licensed for WUInstall, which has this functionality built-in. We added an additional command at the end of the batch file to force WUInstall to scan for updates, and at this point, everything seems to be working properly.

Thanks for reading, and “lforbes,” if you’re out there, thank you for posting your fix.