Exchange Hybrid Migrations: More Than Just a Pretty Face

In my daily support work, I mostly deal with Exchange hybrid migrations (issues) from all our O365 native migration options (hybrid, cutover, staged, IMAP and G Suite). Seeing that no customer ever opened a support ticket if things were OK to tell us they are happy with the migration, yes, I mostly see various issues. Let’s talk about this (large) subject and then transition into troubleshooting!

Exchange hybrid migrations are mailbox migrations between Exchange 2010 (or later) hybrid environment Internet facing Exchange servers on-premises and Exchange Online servers in Office 365. In many cases, those are done via our fantastic Hybrid Configuration Wizard (HCW) tool.
Exchange 2010 SP3 is the minimum required server version for a hybrid deployment (as of this writing) but you can have legacy Exchange servers (2003, 2007) in the environment. You can migrate mailboxes hosted on Exchange 2003 and Exchange 2007 through your Exchange 2010/2013 MRSProxy server to Exchange Online.
Note that during hybrid mailbox migrations, users can still access their mailboxes (those are online moves). An exception was when we had Exchange 2003 coexistence with Exchange 2010 hybrid and the mailbox you tried to migrate was hosted on an Exchange 2003 server; this was called an offline move. More info about this can be found here.

Hybrid migrations are also called Exchange remote moves. As the name implies, this is a move action, meaning we have just 1 (active) mailbox for the migrating user during the whole hybrid migration process and this mailbox is being moved either from:

• Exchange on-premises to Exchange Online (we call this ‘onboarding’ move)
• Exchange Online back to Exchange on-premises (‘offboarding’ move)

All hybrid mailbox migrations, for both onboarding and offboarding, are driven from Exchange Online side, through one of the following methods:

For onboarding, data is pulled from on-premises to the cloud; for offboarding, data is pushed from cloud to on-premises.

Onboard or offboard with PowerShell

Open Windows PowerShell (not Exchange Management Shell) and connect to Exchange Online.

ONBOARD
Run a cmdlet where User is an unique identifier for the migration user like ExchangeGuid but usually it is the display name or SMTP address for the account you want to move. ‘mail.contoso.com’ is the EWS endpoint that has MRSProxy enabled on-premises, and ‘contoso.mail.onmicrosoft.com’ is the routing domain used in Exchange Online. More details here.

New-MoveRequest –Identity ‘User’ -Remote -RemoteHostName ‘mail.contoso.com’ -RemoteCredential $opCred -TargetDeliveryDomain ‘contoso.mail.onmicrosoft.com’

Note that in modern hybrid configurations, the RemoteHostName is pointing to the Hybrid Agent, which is an URL like <GUID>.resource.mailboxmigration.his.msappproxy.net. You can find this on the (Get-MigrationEndpoint).RemoteServer:

 

OFFBOARD
Run a cmdlet where User is an unique identifier for the migration user like ExchangeGuid but usually it is the display name or SMTP address for the account you want to move. ‘mail.contoso.com’ is the EWS endpoint that has MRSProxy enabled on-premises, ‘contoso.com’ is the shared SMTP domain name, and ‘Mailbox Database Name’ is the name of the database you want to move the user to on-premises. More details here

New-MoveRequest –Identity ‘User’ -Remote -RemoteHostName ‘mail.contoso.com’ -RemoteCredential $opCred -TargetDeliveryDomain ‘contoso.com’ -RemoteTargetDatabase ‘Mailbox Database Name’

If you are offboarding a cloud mailbox, make sure that you have a corresponding mail user / remote mailbox object in on-premises Exchange with the cloud mailbox ExchangeGuid and matching SMTP.

Hybrid moves are usually preferred by our customers and admins because of the directory synchronization (users and passwords synced), as well as the Mailbox Replication Service (MRS) benefits like high fidelity in copying the content and metadata stored in the mailbox. With hybrid moves, there is no need to recreate Outlook profiles and re-download OST files.

Hybrid migration is almost completely seamless experience for the end-user with very little downtime. In the final incremental sync stage, MRS locks the source mailbox but this is usually about a couple of minutes in most of the cases. We will touch on final sync lock, in a later part of this blog series (where we will discuss issues with slow migrations).

Unlike IMAP migrations, in hybrid migrations, you cannot currently exclude certain folders from migration (not even the Dumpster); it’s all or nothing.  You can however skip corrupted / bad items or large items.

Starting with Exchange 2010 on-premises environments, users can also have archive mailboxes. The following are archive scenarios currently supported with hybrid moves:

• Moving both primary and archive to Exchange Online as part of the same move request
• Move the primary only to Exchange Online if archive has already been onboarded to EXO
• Move the archive to Exchange Online or back from Exchange Online, while the primary mailbox remains on-premises
• Offboard both primary and archive mailboxes to Exchange on-premises

Note: Moving the primary mailbox to Exchange Online and the archive mailbox to on-premises is not supported at this time. When offboarding an archive back to on-premises, make sure auto-expanding archiving hasn’t been enabled on the cloud archive and the target Exchange on-premises server is minimum Exchange 2010 Server version. More info here.

The 3 supported scenarios for primary mailbox / archive mailbox setup in a Hybrid deployment:

 

Mailbox type	Scenario 1	Scenario 2	Scenario 3
Primary	On-premises	On-premises	EXO
Archive	On-premises	EXO	EXO

 

We recommend hybrid migrations in mostly all non-hosted Exchange on-premises environments (especially where we have minimum Exchange 2010 server version in the environment). Multi-On-premises Hybrid environments with single O365 tenant is also possible and we are making progress to enable support for full hybrid of a single on-premises organization to two or more tenants.

Minimal / express migration options in both classic and modern hybrid are a perfect substitute for traditional cutover or staged (which is a much more challenging migration method). Exchange 2003 and Exchange 2007 servers are very old, out of support lifecycle and if you plan on keeping directory sync, at least for now you need a more recent Exchange server for management purposes of dirsynced Exchange Online mailboxes. We don’t recommend IMAP migrations from Exchange on-premises as this only copies email data and folders, there is more user downtime and no coexistence during migration.

Here is an illustration that summarizes the above:

 

And a quick overview of hybrid migrations / deployments:

Full hybrid

• Enables all functionality available in a hybrid deployment
• Intended for a longer or permanent coexistence.

Minimal hybrid

• Enables minimum functionality necessary to migrate mailboxes and manage them after migration, with limited coexistence functionality.
• Can be used instead of staged migration, needs a minimum of Exchange 2010 SP3 in the environment
• Not recommended if more than 150 mailboxes or 10 servers

Express migration

• Migration option that leverages minimal hybrid
• Facilitates one-time directory synchronization via AADConnect which you are prompted to install during HCW
• Can be used instead of the traditional cutover migration
• Can’t be used if we already have / had DirSync enabled at the tenant level (use minimal)

More info on these hybrid migration options (full, minimal, express) and other hybrid benefits you can check here and here.

As already mentioned, hybrid migrations are Exchange remote moves (the migration endpoint for hybrid migrations is ExchangeRemoteMove type). This implies that we are moving mailboxes between different forests (different Exchange organizations). In contrast, when you do a mailbox move within your Exchange on-premises organization (move a mailbox from one server to another or to a different database on the same server) that is called a local move.

Here is a quick overview of the components and services used in a hybrid remote move:

Microsoft Exchange Mailbox Replication Service (MRS)

Mailbox Replication Service (MRS) is responsible for processing MRS requests. Categories of requests: move requests (local and remote), sync requests (IMAP migrations in EXO), merge requests (EXO Outlook Anywhere migrations like staged and cutover), mailbox import requests (O365 PST import service), public folder mailbox migration requests (public folder migrations); basically anything that has the word “request” in it. Including mailbox restore requests.  
This service is present in the Exchange Online datacenters and allows performing hybrid migrations to and from Exchange Online.
This service is also present on on-premises Exchange 2010/2013/2016/2019 servers and allows local moves or cross-forest moves between other on-premises Exchange forests.
For hybrid moves the communication is between MRS in Exchange Online and MRSProxy in the on-premises Exchange (which is an extension of the MRS on-premises service).  You can find official documentation on hybrid remote moves here. In that article you will find information on how to create a migration endpoint, enable MRSProxy on-premises and how to perform the migration to Exchange online.

Migration Service

This is an orchestration engine on top of MRS. Migration service does not have an independent service running on the server, it is a part of Microsoft Exchange Service Host. This is present in Exchange Online and on-premises Exchange 2013/2016/2019 servers. Migration service invokes the actual MRS cmdlets like New-MoveRequest (hybrid migrations), New-SyncRequest (IMAP migrations), New-MergeRequest (Outlook Anywhere migrations) and is managed using “migration” commands like New-MigrationBatch, Set-MigrationUser, New-MigrationEndpoint etc.

It keeps tenant migration data in special system mailboxes. For example, when we create a migration batch, this service creates multiple messages in the migration arbitration mailbox. You can see this mailbox with Get-Mailbox -Migration cmdlet in Exchange Online PowerShell (reference on the switch here).

EXO tenant admins can do Test-MAPIConnectivity against this migration mailbox:

We can also see mailbox folder stats for this migration mailbox:

Migration Arbitration Mailbox

This mailbox holds the messages that represent migration service objects like migration endpoints, migration users, migration batches, etc.

System mailbox on the Exchange Online cloud database

This is where the MRS job is stored (MRS move jobs, MRS move reports and MRS sync states). You can find more info on how all of this looks, here. A specific cloud database system mailbox in Exchange Online environment might hold jobs for many different tenants (since users from many different tenants might exist on the same database), but the migration service arbitration mailbox will only hold migration service objects for one tenant. We can see this data in SyncMigration and SyncMigrationReports or Settings folders, using the following command:

Get-Mailbox -Migration | Get-MailboxFolderStatistics -IncludeAnalysis -FolderScope nonipmroot | where {$_.FolderPath -like “*Migration*” -or $_.FolderPath -eq “/Top of Information Store/Settings”} | FL folderpath,topsubject*,itemsinfolder 

All migration types in EXO use batch architecture. This allows for easier creation and management of multiple mailbox moves. Typically, hybrid migrations are done through migration endpoints created in Exchange Online but sometimes customers just try New-MoveRequest directly in Exchange Online PowerShell to test connectivity to migration servers or in case they want to bypass migration service and batches process.

If you want to find out more on hybrid migration endpoints and how to troubleshoot them, please check out my other blog posts:

When moving mailboxes through migration batches, the migration service and the components depending on this service play an important part in the hybrid migration process.
Here are the main things that depend on the migration service where the hybrid migrations (managed through migration batches) could be affected:

• If migration service would be stopped in the Exchange Online (this would be an unusual situation)
• If migration arbitration mailbox would be offline or inaccessible (another unusual situation in Office 365)
• If there was a temporary issue in Office 365 that would affect the creation / starting of / management of migration batches (for example a bad parameter / value in New-MigrationBatch or Start-MigrationBatch cmdlet)
• If the credentials of the on-premises migration administrator expired and these are stored on the hybrid migration endpoint in Office 365 (this happens often)
• If the on-premises environment would not allow for connection the IP address of the Exchange Online mailbox server hosting the migration arbitration mailbox of the tenant,  (common mistake made by filtering connections and not allowing all EXO IP addresses to connect to on-premises MrsProxy endpoints).

All the migrations created from the GUI (for example, Exchange Admin Center in Office 365) use the migration service and the migration batch architecture. However, administrator can create a move request directly from PowerShell and this action would bypass the migration service and the dependencies (migration batch / migration endpoint / migration arbitration mailbox). This could be useful when trying to narrow down issues, like for example when migration through EAC fails by throwing out a timeout exception (mailbox server outbound IP is blocked on-premises) but New-MoveRequest done by admin in PowerShell would be successful (the outbound IP of the backend server of the admin being allowed to connect to on-premises).

Here is an illustration of how all of this is connected:

 

When you start a migration via batches, the migration service does several things:

• Create a batch of users as per your inputs: the name of the migration batch, which users are to be migrated in a batch (based on CSV file or selection of people picker in the GUI), settings like bad and large items limit (we don’t recommend setting those anymore as they will be deprecated soon with the new Data Consistency Score model), and settings for manual or automatic startup / completion of the batch.

In the background, the cmdlet New-MigrationBatch <name of the migration batch> will create the batch containing the user identities to be migrated in that batch, ex: john@contoso.com, jane@contoso.com. The migration service will then create new migration user objects for each user identity from the migration batch that you can retrieve later with Get-MigrationUser and Get-MigrationUserStatistics cmdlets.
I wanted to mention here that directory synchronization is required in hybrid deployments and it’s the AADConnect tool that is responsible for synchronizing your users from on-premises Active Directory to cloud Azure Active Directory. This will ensure creation of mail users in Exchange Online with an ExchangeGuid that matches the on-premises mailbox.

The presence of a corresponding mail user in Exchange Online and the ExchangeGuid attribute is vital before starting the hybrid migration. In case you don’t see a mail user object in people picker UI when starting a migration for a specific user in EXO, ensure that the user is being synchronized from on-premises and it has the mail attributes present in on-premises AD where you are syncing from and that those attributes are also synced and brought correctly to Azure AD/ Exchange Online. Check Get-MsolUser /Get-AzureADUser  (MSODS/Azure AD PowerShell) and Get-MailUser and Get-User (EXO PowerShell) before starting a hybrid migration and make sure the user is there, with the correct alias, proxy addresses and ExchangeGuid is stamped on it.

Having a mail user in Exchange Online without the ExchangeGuid for the on-premises mailbox will cause a number of issues during migration process. First, the migration process won’t be able to find the on-premises user to migrate and will thus fail. Second, if you assign an Exchange Online license to the mail user without ExchangeGuid, then you would end up with duplicate mailboxes for the same user because the cloud service is unaware that the user has a mailbox located on-premises. If the ExchangeGuid is correctly synced, when you assign the EXO license to the synced mail user with corresponding on-premises mailbox, you would see the expected warning on user properties:

This user’s on-premises mailbox hasn’t been migrated to Exchange Online. The Exchange Online mailbox will be available after migration is completed.

We recommend assigning an Exchange Online license before starting the hybrid migration so that you don’t forget to assign the license after migration and risk the mailbox loss after 30 day grace period. But again, assigning the EXO license on an incorrectly provisioned mail user will get you into the situation when the migration user would have a mailbox both in Exchange on-premises and EXO.  If you get yourself into this situation, please follow this. If you are interested on how to provision EXO mailboxes in a hybrid deployment, please see this blog post.

Coming back to migration batches: hybrid migrations and G Suite migrations are the only ones where migration batches can be completed (Complete-MigrationBatch). And you should be aware that there is a 100 batches limit to help maintain the responsiveness of the migration service.
Migration batches are associated with migration endpoints. In hybrid deployments the migration endpoint is typically created automatically by HCW (based on the migration admin credentials inputted into the wizard) or by admin when starting a migration.

• Migration Service then invokes cmdlets specific for the migration type chosen (hybrid, cutover, staged, IMAP/G Suite), example New-MoveRequest john@contoso.com -Remote for a hybrid migration (Exchange remote move).
• The service then monitors the status of the migration user, periodically sending emailed reports to the configured email addresses to inform of progress.

The remote mailbox move process: combining the two services in practice

The following is an overview of the hybrid mailbox move process. Understanding the process at a basic level is important so you can become more proficient in identifying where the issues may be.

1. The New-MoveRequest cmdlet prompts MRS on the mailbox server in Exchange Online where the mailbox is located or will be located.  Even though I’ve already mentioned this, I feel like underlining again the importance of this prerequisite as it saves a lot of time and avoids issues: Before you can create an onboarding move request (New-Moverequest), a corresponding mail-enabled user needs to exist in Office 365 for the mailbox you plan to move. This will have been created by AAD Connect tool in the directory synchronization phase. It is VERY important to make sure that the corresponding mail users are found in Exchange Online and that they were provisioned correctly, meaning they have the matching attributes like ExchangeGuid from on-premises mailboxes, they have a secondary smtp address matching user@tenant.mail.onmicrosoft.com and that we have accepted domains verified in EXO for their proxy addresses domains. If this is not done correctly, the migration will fail at validating user phase or the migration service won’t be able to inject the move request.
If all checks out, the new mailbox move request is Queued in Exchange Online and processed by the datacenter. The cmdlet (New-MoveRequest) updates the Active Directory information (attributes like msExchMailboxMove* e.g. msExchMailboxMoveRemoteHostName) and the system mailbox on the cloud database, depending on whether it is a push or pull move.

2. Now, an MRS instance is ready to act upon the request. To initiate the move, MRS in the Exchange Online forest communicates through MRSProxy in the on-premises forest.
The MRS server in EXO pulls the mailbox data from the mailbox server through the MRSProxy server on-premises to the mail-enabled user in Exchange Online. At this point, the status is In Progress.

3. When the mailbox move is almost complete (95% completion), MRSProxy locks the mailbox on-premises for a short time while final synchronization is completed. The status is still In Progress (completion in progress).

4. In Exchange Online, MRS converts the mail-enabled user to a mailboxIn the on-premises forest, MRSProxy converts the mailbox to a mail-enabled user of type ‘remote mailbox’ (Enable-RemoteMailbox), and the source (on-premises) mailbox is soft deleted. At this point, the status is Completed.

Note that when you migrate a mailbox hosted on Exchange 2007  or an Exchange 2010 pre-SP1 through a hybrid migration, the on-premises mailbox is permanently deleted (no soft-deletion here).

You can view soft-deleted mailboxes with the following cmdlets in on-premises Exchange Management Shell (EMS):

Get-MailboxDatabase | Get-MailboxStatistics | where {$_.DisconnectReason -eq ‘Disabled’ -OR $_.DisconnectReason -eq ‘SoftDeleted’} |FT displayname, mailboxguid, database,disconnectdate
Get-MailboxDatabase | FT name, guid, mailboxretention
Get-RemoteMailbox <user> |FL

Other AD updates during completion:

• The LegacyExchangeDN attribute is restamped to /ou=External
• The original LegacyExchangeDN is moved to an x500 proxy address in order to preserve the ability to receive replies to old emails
• A RemoteRoutingAddress / TargetAddress is set on the user object that points to the coexistence domain tenant.mail.onmicrosoft.com (in EMS: Get-RemoteMailbox <user> |FL RemoteRoutingAddress). This is needed for routing mail flow and Autodiscover requests from on-premises to cloud.

5. Optionally, the administrator clears the move request/ removes migration batch and with this process, the move information is cleared from Active Directory and from the system mailbox. Until the move request information is cleared, you can’t move the mailbox again. Sometimes, clearing the move request in EXO doesn’t succeed in clearing the AD attributes on-premises (for example if during the completion, the on-premises migration admin credentials expired).

When the migration is initially kicked off (step #1) , the move request may be queued and potentially throttled by Exchange Online for a number of reasons. One such reason is to enforce the Max Concurrent Migrations configured for the migration endpoint, or to back off when the on-premises MRSProxy declares that it is overloaded. It is important to understand that some delays are considered normal; the mailbox move process is considered to have a lower precedence than things like client connectivity and mail flow. Some of this information may be found in the MoveRequestStatistics logs and will be discussed in a later part of this blog series.

Mirela Buruiana