Create Your Own Adaptive Retention Policies with PowerShell and Azure Automation

Avoid High-End Licenses with DIY Adaptive Retention Policies

In my last article covering adaptive scopes for Microsoft 365 retention policies, I covered how to use Azure AD custom attributes to identify the set of accounts for retention processing. Apart from taking too long to evaluate and report the set of accounts found by the scope query, adaptive scopes work well and are a good way to make sure that Microsoft Purview retention processing applies to specific sets of accounts automatically without ongoing updates by an administrator.

The downside of adaptive scopes is their requirement for Office 365 E5 or Microsoft 365 E5 compliance licenses. Microsoft very much wants to maximize the amount of revenue generated from its cloud user base, perhaps as a result of slowing Office 365 user growth. The upshot is that any feature that includes automatic processing, especially in the compliance area, usually requires high-end licenses.

Static Retention Policies

Microsoft 365 retention policies with static scopes are available to Office 365 E3 tenants. A static scope means that administrators are responsible for managing the set of locations processed by a policy. A location can be a mailbox, site, group, team, or OneDrive account.

Sometimes, no management is needed, as in the case when a retention policy applies to all locations of a certain type (like all mailboxes or all sites). In other circumstances, making sure that the set of static locations specified in a policy is accurate can take considerable effort.

PowerShell and Azure Automation Deliver Updates for Adaptive Retention Policies

The thought occurred to me that it should be possible to build a form of do-it-yourself adaptive retention processing with a combination of a PowerShell script and Azure Automation. The combination replicates what adaptive scopes do without the need for high-end licenses. In this scenario, the PowerShell script:

• Identifies the set of locations for the retention policy to process. Adaptive scopes query Azure AD to find this information. Our script can do the same.
• Checks the locations already specified in the retention policy to calculate locations that should be removed and those to be added.
• Updates the retention policy with the new locations.

Azure Automation runs the script on a scheduled basis to detect and apply changes to the retention policy. The recent addition of support for managed identities by V3.0 of the Exchange Online management module makes this process easier, even if the cmdlets that run against the compliance endpoint still require credentials to connect (Microsoft is working to fix this problem and a fix might be available when you read this article).

An Example Implementation

To prove that the principal works, I wrote an Azure Automation runbook (a version of a script to run in the Azure Automation environment). Adaptive scopes can process users and groups or SharePoint Online Sites. For this demonstration, the script processes user accounts by updating the Exchange Online mailboxes and OneDrive for Business accounts owned by the user accounts found by the query.

To begin, I created a new static retention policy with a static scope and defined one location for mailboxes and OneDrive accounts (Figure 1). Although this might seem like a strange step, it’s important to do this and not target the policy at all available locations, A policy with All shown for a location looks for and processes all locations. For instance, if you choose to process All mailboxes, the policy processes all user, shared, and inactive mailboxes.

Microsoft Purview treats retention policies differently when they process all available locations for a workload (like all mailboxes). The Set-RetentionCompliancePolicy cmdlet used by the script can only update locations if the policy is set up to process selected locations. In other words, the Set-RetentionCompliancePolicy cmdlet can’t switch a policy between all-in and selective processing.

Writing the Script

With a target policy to work against, we can go ahead and find some mailboxes and OneDrive accounts to process. The script does the following:

• Connects to the Microsoft Graph PowerShell SDK with a managed identity to run the Get-MgOrganization cmdlet to find the default domain for the tenant. We need this to calculate the URLs for the target OneDrive for Business accounts.
• Connect to Exchange Online with a managed identity.
• Connect to the compliance endpoint using account credentials stored in Azure Key Vault. The need to use stored credentials will no longer be necessary when Microsoft upgrades the compliance cmdlets to use managed identities (or fixes certificate-based authentication).
• Use the same account credentials to connect to the SharePoint Online management endpoint.
• Find the set of target mailboxes. This example uses the same custom attribute to mark mailboxes as in the previous article.
• Find the set of OneDrive for Business accounts in the tenant and store the owner name and URL in a hash table.
• Loop through the set of mailboxes to check the user principal name for each mailbox against the OneDrive hash table. If a match is found, we know that the OneDrive account belongs to the mailbox owner, so we store it in a list.
• Retrieve the set of current locations in the policy using the Get-RetentionCompliancePolicy cmdlet.
• Check what mailboxes are not in the current location set and create a list of mailboxes to add to the policy. Do the same for OneDrive accounts.
• Check if any mailboxes and OneDrive accounts are within the scope of the policy but shouldn’t be. If we find any, add them to a list of locations to remove from the policy.
• Call the Set-RetentionCompliancePolicy cmdlet to add locations that should be in the policy and remove those that should not.
• Call the Get-RetentionCompliancePolicy cmdlet to report the set of locations now covered by the policy.

I’ve never seen the code Microsoft uses to implement active scopes for retention policies, but I’m certain that much the same kind of processing occurs to figure out the set of locations that a policy should cover. I’m sure that the Microsoft code is more elegant and precise than mine (and not written in PowerShell), but the point is that the code works and does the same job as an adaptive scope does.

Be aware that a compliance policy supports a maximum of 1,000 individual mailboxes. More importantly, the limit is lower (100) for SharePoint Online sites and OneDrive for Business accounts. Testing and error handling for scripts used in production need to handle these limits. To get around the limits, you can create multiple policies with the same retention settings targeted at different sets of mailboxes, sites, and accounts.

Running the Script

You can download the script from GitHub and test it for yourself. The script runs in Azure Automation. Some changes to the authentication section of the code will make it run interactively.

Figure 2 shows the results of the script after running in the test pane of an Azure Automation session. The output shows object identifiers for the mailboxes to add and remove, while URLs are used for OneDrive for Business accounts. These are the values used to identify locations internally. If you wanted to, it would be easy to change the outputs to show the names of the accounts. However, as the script runs as a scheduled Azure Automation task, no one will probably look at the output unless something goes wrong.

Figure 3 shows the properties of the retention policy after the script runs. Where previously there was just one location noted for both Exchange Online and OneDrive for Business, now the policy covers 24 mailboxes and 23 accounts. The discrepancy between the two numbers is due to a new account that hasn’t yet signed into OneDrive.

The same principal explained here could extend to create an adaptive retention policy for Microsoft 365 Groups or SharePoint Online sites. It’s just a matter of PowerShell programming.

DIY is Not Always Possible

Recreating a Microsoft 365 feature with DIY code isn’t always possible. In this instance, it’s reasonably straightforward to create a custom form of adaptive retention processing. The combination of PowerShell and Azure Automation effectively replicates what Microsoft delivers in adaptive scopes.