Exchange

Nested Groups in Azure AD and Exchange 365

Posted by James F. Prudente on May 9, 2023
Posted in: Active Directory, Exchange, Office 365. Tagged: . 2 Comments

As in a lot of environments, we sync our on-prem AD to Azure AD, and as such, on-prem security groups are used to grant access to various cloud resources. For us, this includes shared calendars in Exchange 365.

We received a ticket that a few users were not getting the proper level of access to a shared calendar, and my initial investigation did not turn up anything that would explain the problem. Permissions were granted to a sync’d AD group, which itself contained two nested AD groups. Group nesting has always been a bit quirky in M365, so that was the obvious place to check, but users in the first nested group had the proper access. Only users in the second group were missing permissions.

I checked Azure AD and was able to confirm that all members of both groups were properly being placed in the parent group, so clearly the nesting itself was not the issue and there had to be something else going on. I started looking at the differences between the two nested groups and found that the working group was a mail-enabled security group, while the non-working group was not mail-enabled. Turns out that was the problem.

Evidently, Exchange 365 ignores group membership unless the relevant group is mail-enabled. It’s been a while since we had on-premise Exchange but I do not recall this being the case there. In any event, I added an e-mail address and proxy address to the problem group (in AD), and once things sync’d up to the cloud, the users had the proper permissions.

I was not able to find any documentation for this. Hopefully this post saves someone else the trouble of figuring it out on their own. If it does, or you have anything else to add, I’d love to hear from you.

Managing Mail-Enabled Security Groups in O365

Posted by James F. Prudente on September 7, 2018
Posted in: Active Directory, Exchange, Office 365, PowerShell. 3 Comments

Office 365 certainly has its plusses and minuses, and one of the areas that clearly falls in the latter category is how O365 handles mail-enabled security groups.

Consider the following scenario:

  • You have an on-premise Active Directory sync’d to O365/Azure via ADSync.
  • You have an on-premise security group that you mail-enable, creating an e-mail address @yourdomain.com.
  • This mail-enabled group syncs to O365, and then exists in Exchange Online.
  • You then mail-disable the group on-premise and wait for the change to sync to Exchange Online.

Logic would dictate that once the mail-enabled attribute was removed from the group, it would disappear from Exchange Online. Of course it’s not that simple.

What you will instead find is that the security group still shows in Exchange Online, however it now has a proxy address @yourdomain.onmicrosoft.com. It will also continue to be visible in both the online and offline address books. If you are trying to move groups to the cloud and/or cleanup your distribution lists, this can cause confusion.

The fix fortunately is easy, but it’s an extra step that shouldn’t be necessary IMO. Credit to Tim McMichael’s TechNet post for the details.

The process to properly remove a mail-enabled security group from Exchange Online is as follows:

  • On-Premise, remove the mail-enabled attribute from the group using the Disable-DistributionGroup PowerShell command.
  • On Exchange Online, find the group via Get-MsolGroup –SearchString “Group Name”.
  • Once you’ve confirmed you have the right group, remove it via Get-MsolGroup –SearchString “Group Name” | Remove-MsolGroup

This will finally remove it from Exchange Online and your address books.

EDIT: There is a major caveat here to be aware of. If you are using mail-enabled security groups to grant permissions to shared calendars, you CANNOT remove the mail attributes or you will break access permissions. In that situation I would advise simply hiding the list from the address book.

VSS Snapshots Fail on Server 2008 or 2008 R2

Posted by James F. Prudente on March 8, 2017
Posted in: Exchange, Windows Server. Leave a comment

Recently our Exchange 2013 server – running on Windows Server 2008 (not R2) – started having backup failures. We use FalconStor DiskSafe for backup, which, like most (all?) Exchange-aware backup application uses VSS to snap the server.

During the investigation with FalconStor support, we discovered VSS was timing out during the verification of the snapshot, and running “vssadmin list writers” showed a number of the VSS writers in a failed state. No updates had recently been applied to the server, nor had any configuration changes been made recently, so there was no obvious potential cause of the problem.

FalconStor support was able to point us to this link at IBM. While not an exact description of our problem, the article linked to a Microsoft utility, DevNodeClean. The description for that utility is:

“On a computer that is running Windows Server 2003 or a later version, a storage device that is connected by using a fiber channel or by using the iSCSI protocol may be connected for only a short time. When a storage device is connected, Windows creates registry information for the device. Over time, the registry may contain many entries for devices that will never be used again. This utility can be used to remove this information from the registry.”

Interesting. Searching a bit more I came across a 3rd party complied version of DevNodeClean, which has more information about the problem it resolves.

I can’t find the link now, but I also came across a page that suggested reviewing the size of the registry itself. The actual registry files are located at c:\windows\system32\config. The SYSTEM file, which represents HKLM\SYSTEM, was in this case over 1.5GB! Checking a few other servers of similar age showed SYSTEM files of 50MB or less in most cases.

By now, I was pretty confident this was our issue. I ran DevNodeClean (the Microsoft provided one), which took – no joke – six days to complete. As it continued to delete unused devices from the registry, our backups began running again intermittently. After about five days of running the utility, all backups began working properly.

It seems what happens is that any backup product that calls VSS – FalconStor, Veeam, Microsoft DPM, etc. – causes a registry entry to be created for every single VSS snapshop. I believe the same happens when using Hyper-V snapshots. These registry entries are not cleaned-up, and over time, they bloat the registry to the point where it can essentially break VSS. In our case, we were snapping two LUNs every hour, so over the course of a few years of the server’s life, there were thousands of orphaned registry entries.

There is supposedly a hotfix for 2008 R2, but it’s not clear if that fixes the problem, and in any event it’s not available for 2008.

It seems the best way to deal with this is to run DevNodeClean as a scheduled task on a regular basis.

Unfortunately, the registry cannot be compacted online, so we will need to schedule a maintenance window to actually shrink the bloat.

Fixing Missing Address Book Entries in O365

Posted by James F. Prudente on October 6, 2014
Posted in: Active Directory, Exchange, Office 365, Windows Server. Tagged: , , , . Leave a comment

Recently I’ve been working to get a full O365 implementation setup to roll-out to our students, and while we’re not doing student e-mail just yet, I wanted to make sure everything is working so when we are ready, we just have to add the Exchange licenses. One of the quirks I ran into is a few users were not listed in the Office 365 address book. We do not have a hybrid Exchange deployment, but because our AD is sync’d to O365, we should have a full GAL in the cloud accessible to be searched by any Office 365 user.

After going through the usual online search and not turning up anything too helpful, I decided to compare O365 data for two users: one who was showing up in the GAL, and another that was not. I piped the output from Get-MSOLUser to a separate text file for each user, then used WinMerge to compare them.

Get-MSOLUser –UserPrincipalName UPN1 | fl > user1.txt

Get-MSOLUser –UserPrincipalName UPN2 | fl > user2.txt

The one attribute that stood out as different was CloudExchangeRecipientDisplayType, which was set to “-2147483642” for the user that was in the GAL, and “1073741824” for the user that was not. I started researching that attribute and found this very helpful post. The poster of that Article, Declan Conroy, was trying to trick O365 in such a way as to facilitate an easier migration from on-premise Exchange to the cloud. It wasn’t an exact fit for my situation, but it did make clear that “CloudExchangeRecipientDisplayType is not available as an attribute in AD, [but] is exported…with the value from the incoming msExchRecipientDisplayType attribute.”

That led me to look for more info on msExchRecipientDisplayType, and I found this page listing the meaning of the various values for both that attribute and msExchRecipientTypeDetails.

At this point it still wasn’t clear to me if the discrepancy in CloudExchangeRecipientDisplayType was causing the missing GAL entry, and because the only way to change that attribute in the cloud was to change msExchangeRecipientDisplayType in my local AD, I wanted to investigate more.

I decided to search for other users with 1073741824 as their CloudExchangeRecipientDisplayType:

    Get-MSOLUser | Where-Object {$_.CloudExchangeRecipientDisplayType -eq 1073741824}

The results from this search seem to confirm that this is the expected value for users who do not have on-premise mailboxes.

Some more digging turned up another disparity between accounts in the GAL and those that were not. I ran this command:

    Get-User –Identity displayname | select RecipientType, RecipientTypeDetails

I found three distinct results:

RecipientType

  

User

These recipients were not in the GAL

MailUser

User with on-premise mailbox, in GAL

UserMailbox

User with O365 mailbox, in GAL

 

This provided further evidence that the incorrect CloudExchangeRecipientDisplayType, which was controlled by the on-premise msExchRecipientDisplayType attribute, was the source of the problem. So now to fix it…fortunately I had already found the answer in a post I linked earlier.

Using the attribute editor in Active Directory Users & Computers I made three changes to each affected AD account:

Attribute

Change To

msExchRecipientDisplayType

6

msExchRecipientTypeDetails

128

targetAddress

User’s e-mail address

 

Note that to access the attribute editor you must have advanced features enabled, and for some reason that tab does not show if you perform a search for the user. You’ll need to manually find them in AD and pull up their properties there instead.

You then need to perform a sync of your on-premise AD to the cloud. After doing so, RecipientType and RecipientTypeDetails both showed “MailUser,” and CloudExchangeRecipientDisplayType showed “-2147483642,” both of which were expected values for a working on-premise mailbox. A quick check of the address book showed the users were now listed in the O365 GAL.

You’re still not done though. If you leave the settings like this you will break your on-premise mailbox. (Why is it you only discover this stuff late on a Friday night, when you’re trying to relax?) You must change msExchRecipientDisplayType, msExchRecipientTypeDetails, and most importantly targetAddress back to their original values (1073741824, 1, and not set, respectively) in your on-premise AD. This will keep your mailbox working but the change to the cloud user type will remain. All set!

In speaking with a colleague recently I remarked that 90% of what you do in Office 365 is easy and works well, but the other 10% is a real source of pain. As we’ve seen here, much of that 10% comes from the fact that it’s not clear what many O365 attributes represent, and even when it is, those attributes are often not directly editable by administrators. Hopefully that situation will improve in time.

[PublicFolderDatabase] is pointing to the Deleted Objects container in AD

Posted by James F. Prudente on April 16, 2014
Posted in: Exchange. 4 Comments

After installing Exchange 2013 SP1, I began reviewing the server’s Application log for errors or warnings. There were a large number of Event ID 2937, from MSExchange ADAccess, and while the exact error text varied a bit from event to event, they all indicated that an Exchange property was pointing to a deleted object in AD. The process itself can vary, as can the property in question. In our case it was [PublicFolderDatabase] as shown below.

While researching the issue I was able to find a bit more information about properties other than [PublicFolderDatabase]; in some cases this error seems to indicate missing or bad data that must be corrected for Exchange to function properly. Our specific error wasn’t causing any operational problems but it was littering the Application log.

What I found strange after doing some research was that the PublicFolderDatabase property is deprecated in Exchange 2013 and thus shouldn’t be causing a problem. Nonetheless, the data was still there and I needed to remove it. Unfortunately with the property deprecated, Set-MailboxDatabase could not be used to set PublicFolderDatabase to $null; attempting to do so resulted in “Cannot validate argument on parameter ‘PublicFolderDatabase’. The argument is null or empty…”

The other info I could find out there called for using ADSI Edit to remove the property, but try as I might I couldn’t find the PublicFolderDatabase property where the App log said it should be. Finally I stumbled on this post which, while not pertaining specifically to Exchange 2013 identified the actual attribute name as “msExchHomePublicMDB”. I still couldn’t find it though until I read this comment, which explained the problem: I had been viewing the properties of the “notebook” object, rather than the “folder” object. (See screenshot below)

Once I opened the properties of the folder object, sure enough I found “msExchHomePublicMDB” and was able to clear it out. Once replication occurred, the errors stopped. Simple fix once you know where to look.