Digging Into Hybrid Migration Move Report Data

To continue my hybrid migration blog series (see part 1 here) – let’s talk about migration move reports. We will need this knowledge when we get to later parts of the series (and start troubleshooting more).

When troubleshooting failed or slow migrations, it is best to export the move reports to XML files. If you are working with Microsoft support, you can then send those files to us. If we have the move request statistics with -IncludeReport and -DiagnosticInfo switches for the affected users, we’d typically have enough to troubleshoot hybrid migrations (assuming that the move requests were created for the users we are troubleshooting).

What if move requests are not being created?

If you started the hybrid migration via migration batches and move requests are not there because they weren’t created by migration service for some reason, you would send us the migration user statistics XML and eventually the migration batch XML.  We would also require you to run New-MoveRequest command manually from PowerShell depending on the migration scenario you are doing (onboarding or offboarding) and see if you get an error message when manually invoked by the administrator (the exact commands for onboarding / offboarding move requests are found in Part 1 of the blog).

Here is a quick summary of main reasons why migration service wouldn’t be able to inject the New-MoveRequest (and thus produce the move report):

• Migration user fails validation, for example:
  • ExchangeGuid not synced from on-premises mailbox to corresponding cloud mail user, error referenced here
  • Secondary SMTP address user@tenant.mail.onmicrosoft.com is not synced from on-premises mailbox to corresponding cloud mail user, error referenced here
  • Primary or secondary SMTP address on the on-premises mailbox contains an SMTP domain that is not an accepted domain in Office 365 (the domain hasn’t been verified in Office 365 or it can’t be verified (for example a domain is a non-routable domain like user@contoso.local), error referenced here
• The user mailbox you are trying to migrate to Office 365 has multiple ComponentShared mailboxes in Office 365 and during the merging process we fail to find the recipient ExchangeGUID of a specific ComponentShared mailbox (the error message is similar to #1 above with the difference that the ExchangeGuid doesn’t belong to the user mailbox being migrated but rather belongs to the Exchange Online ComponentShared mailbox location associated with the corresponding cloud mail user for your on-premises mailbox) . If you find yourself in this scenario, please provide us with the error message containing the affected ExchangeGuid and result of Get-MailboxLocation -User <Identity> |FL ran in Exchange Online PowerShell, against the user you are failing to migrate.
• On-premises environment is filtering connections from Office 365 and is not allowing specific Exchange Online IP addresses ranges which includes the outbound IP of the migration service server. Ensure you allow all EXO IP addresses from here.

When move requests are being created

Suppose you did have a move request created (migration service or the admin managed to successfully inject the move request for the user with New-MoveRequest) but the move request is in a failed state. You would then typically retrieve the move request statistics with -IncludeReport switch for a better understanding of the cause of the failure.

There is another switch called DiagnosticInfo switch; it’s use is less common than IncludeReport and is usually used by support people. It accepts 3 arguments: showtimeline, showtimeslots and verbose; showtimeline and showtimeslots are useful when we have slow migrations because we want to see where the requests spent most of the time and useful to understand why it is slow. Verbose argument can be also useful in failed migrations, especially when we don’t have a report for the move/ migration user.
A move request can have the following statuses and you can use –MoveStatus parameter to filter the move request based on a specific status, reference here. I will list the main statuses of move requests, with a brief explanation of each:

• AutoSuspended – the move request is automatically suspended by MRS. This happens if you manually create a move request with the -SuspendWhenReadyToComplete flag or use this parameter on an existing move request and the move request has completed the initial sync stage.
• Completed – the move request completed successfully.
• CompletedWithWarning – the move request is completed but it encountered a failure during the move, a failure which didn’t fail the migration. This needs your attention. We will get deeper into this in a later part of this blog series.
• CompletionInProgress – the move request is about to complete (final sync stage).
• Failed – the move request encountered a permanent failure that is not retriable automatically. Failed move requests will be discussed in Part 3 of this blog series.
• InProgress – the move request is in progress. If you suspect that the move is progressing slowly, Part 4 of this blog series will cover that.
• Queued – the move request is being queued in the MRS queue and is waiting to be processed.
• Suspended – the move request is manually suspended (for example, Stop-MigrationBatch/MigrationUser or Suspend-MoveRequest initiated by the admin).
• Synced – the move request has completed the initial sync stage and is waiting for manual or automatic resume (incremental or final sync stage).

As promised earlier, here are the cmdlets to export migration and move reports to XML files:

Export migration user statistics for one migration user to an XML file

• Use whenever you initiate a hybrid migration through batches
• Use when a move request fails to be injected (you have nothing returned in Get-MoveRequest for the user, meaning no move request available for the user identity) 

Get-MigrationUserStatistics <User> -IncludeSkippedItems -IncludeReport -DiagnosticInfo Verbose | Export-Clixml C:tempAffectedMigUser1Stats.xml

Export move request statistics for one user to an XML file

• This only applies to hybrid remote moves
• Use when you have a Get-MoveRequest for that user identity

Get-MoveRequestStatistics <User> -IncludeReport -DiagnosticInfo “showtimeslots, verbose, showtimeline” | Export-clixml C:tempEXO_AffectedHybridMoveRequest1Statistics.xml

Export all migration batches info to an XML file

Get-MigrationBatch -DiagnosticInfo Verbose -IncludeReport | Export-Clixml C:tempEXO_ALL_Batches.xml

Export one migration batch info to an XML file

Get-MigrationBatch “<Batch Name>” -DiagnosticInfo Verbose -IncludeReport | Export-Clixml C:tempEXO_AffectedBatch.xml

Some of you might be in a situation where the migration completed fine and you have removed the move request but now, an end-user calls you saying that she/he lost permissions on a mailbox you just migrated (and already deleted the report for). Fortunately, you can still try to retrieve the move report and move history with Get-MailboxStatistics command executed on premises where the mailbox was moved to (and the mailbox is hosted currently – Exchange Online or Exchange on-premises). References for the command to use can be found here and here.

Other exporting to XML, you can also store the report in a variable, for example $stats. This is useful especially when you want to investigate by yourself (before logging a support case, for example):
$stats = Get-MoveRequestStatistics <user> -IncludeReport -DiagnosticInfo verbose

Note that if you run the command without those 2 switches, you won’t have the report or DiagnosticInfo available, but only the actual statistics, example:

You can use |FL to see more info on the statistics, example here:

Next I will list some of the main things that can be found in just the statistics of a move request, which I find useful:

• Information related to identifiers of the user mailboxes (primary or archive mailbox) being migrated: MailboxIdentity, DistinguishedName, Alias,DisplayName, ExchangeGuid, ArchiveGuid, Identity
• Information related to the state of the move and its completion settings: Status, StatusDetail, SyncStage, PercentComplete, CompleteAfter, SuspendWhenReadyToComplete, Message, FailureType,FailureSide, LastFailure
• Information related to the directionality of the move and if online or offline move: Direction, WorkloadType, IsOffline
• Information related to the size of the mailboc being migrated: TotalPrimarySize, TotalPrimaryItemCount, TotalArchiveSize, TotalArchiveItemCount, TotalMailboxSize, TotalMailboxItemCount
• Information related to bad items, large items and Data Consistency Score: BadItemLimit, LargeItemLimit, AllowLargeItem, BadItemsEncountered, LargeItemsEncountered, MissingItemsEncountered, DataConsistencyScore, DataConsistencyScoringFactors, LastSkippedItemEncounteredTimestamp, SkippedItemApprovalTimestamp
• Information related to the batch and migration endpoint used (when we initiate hybrid migration through batches): BatchName, MigrationMailboxGuid, SourceEndpointGuid, MigrationBatchGuid, MigrationUserGuid
• Information related to source or remote environments: SourceVersion, SourceDatabase, SourceServer, SourceArchiveDatabase, SourceArchiveVersion, SourceArchiveServer, RemoteHostName, RemoteCredentialUserName, RemoteDatabase, RemoteDatabasename, RemoteArchiveDatabase, RemoteArchiveDatabaseName
• Information related to the target environment and somains: TargetVersion, TargetDatabase,TargetServer, TargetArchiveDatabase, TargetArchiveServer, TargetArchiveVersion, TargetDeliveryDomain, ArchiveDomain

Information related to duration and timestamps:

Now, let’s talk about things we see in the actual report of a move request statistics ($stats.Report) in PowerShell (and with MRS Explorer GUI tool) so that you know what to check when you’re stuck with a hybrid move.

Note: you need to have a move request in place for the user and get the move request statistics with IncludeReport switch in order to have the report available for analysis.

Suppose we stored the move request statistics in $stats variable and have the report available in $stats.Report. We can simply run $stats.Report to dump the entire report (not practical) or check specific sub-parts of the report.

I will now list the main report components and a brief explanation of what type of info you will find here:

• .Entries (all report entries, everything from the report is logged here; think of it as a timeline of the move operation)
• .DebugEntries (“hidden” entries, these are for example useful when troubleshooting in-depth permissions loss during move or want to check the values changed in Set-MoveRequest)

DebugEntries is different from Entries in data contained and as well as localized string language; in report, Entries will be in lof the admin whereas DebugEntries are in English (these are mostly used by Microsoft folks). DebugEntries is usually used in troubleshooting permissions loss (folder ACL before and after the move, mailbox security descriptors (SD) and calendar folder ACL for F/B issues)

Commands to get Entries and DebugEntries linked together and copied to clipboard::

$e =@()
$e += $stats.Report.Entries
$e += $stats.Report.DebugEntries
$e | sort CreationTime | % { [string] $_ } | clip

Command to get the first entry related to a corrupt item – this is also seen in report.baditems[0]:

$stats.Report.Entries | where { $_.LocalizedString -match ‘Corrupt’ } | Select -first 1 | % { [string] $_}

• .Failures (these are admin exposed failures that were encountered during the move; the failures are normally grouped in transient and permanent ones. We will talk about this more in a later part of this blog post series, where we will discuss failed migrations)
  We can output these with a | group failuretype and show overall details or we can output it with a | Select -Last 1 or [-1] to show you the last error in the report or with a | Select -First 1 or [0] to show you the first error in the report. We are mostly interested in Timestamp, SourceSide, DataContext, StackTrace and InnerException
• .InternalFailures (“hidden” failures, used mostly by migration engineering team, usually seen in mailbox locked scenarios; you can format the output the same as regular failures described above)
• .BadItems (corrupted items encountered, most common ones are orphaned ACLs with unresolved SIDs)
• .LargeItems (large items encountered, in MRS migrations limit is 150MB)
• .Connectivity (server versions and names; think of this as a timeline of the connectivity during a move operation)
• .SessionStatistics (statistics of the move , like source and target average latency time)
• .Mailbox verification (item statistics before and after the move on source and target; it’s only available when status is Completed). A quick command to see the folders and their properties where the number of items from source are different from the ones on target: $stats.Report.MailboxVerification | where { $_.Source.Count -ne $_.Target.Count }
• .SourceMailboxBeforeMove.Props together with .TargetMailUserBeforeMove.Props (for example, in onboarding scenario: on-prem mailbox properties and Office 365 mail user properties before the move; only available when status is Completed)
• .TargetMailboxAfterMove.Props together with .SourceMailUserAfterMove.Props (in the same example, in onboarding scenario: on-prem mail user properties and Office 365 mailbox properties after move; only available when status is Completed)
• .SourceMailboxSize (the size of the source mailbox)
• .TargetMailboxSize (the size of the target mailbox)
• .SourceArchiveMailboxSize (the size of the source archive mailbox)
• .TargetArchiveMailboxSize (the size of the target archive mailbox)
• .SourceThrottles  (shows reason for why/if a move was throttled on the source side and  for how long)
• .TargetThrottles (shows reason for why/if a move was throttled on the target side and for how long)

Using MRS Explorer

If you are not super skilled with PowerShell and would like to choose a more user-friendly interface to analyze the move report, you can use the MRS Explorer utility (please note that this is not an officially supported tool).

You will have to download the script located here.

1. After you have downloaded MRS_Explorer.ps1 script to your local drive, run it from any PowerShell client as shown below:

2. From the interface that pops up, you can import one of the XML migrations reports that you have already saved on your machine or you can connect directly to Exchange Online by selection the option “Query Exchange”.

3. Once the report has been imported, you can view the details of each property from the migration report. We can view more insight by expanding the “Report” tree, for example the details of SessionStatistics:

4. We can also see the information about Data Consistency Score for example:

This concludes part 2 of this blog series. Onto failed migrations next!

Special thanks to the army of migration specialists that reviewed and contributed to this post: Angus Leeming, William Rall, Brad Hughes, Chris Boonham, Ben Winzenz,  Cristian Dimofte, Nicu Simion, Nino Bilic, Timothy Heeney.

Mirela Buruiana