In this post, we are going to look at the Mailbox Details log file that can be used when troubleshooting Server Side Synchronization and App for Outlook scenarios. Thanks again to Cody Dinwiddie on our Dynamics 365 Support team for helping put this information together.

Now, if you don't know, Server-Side Sync is processed by the Async Service, which means it can sometimes be difficult to find enough detail to troubleshoot ACT and Email synchronization issues. Mailbox alerts is always a great place to start but it can really only give you a few pieces of information, such as if a Test and Enable was successful or what errors the mailbox is running into. However, this will not show me other key information, such as when the next sync cycle is for the mailbox. In most cases, this is the information I am looking for and it is one of the most common questions I receive.

To find additional information, the mailbox record does have a way to get this data by selecting Download Mailbox Details on the ribbon of the appropriate mailbox record.

Once you click to download this .log file, open it up with a text editor like Notepad. Note: For a Dynamics 365 Online instance, all times will be in UTC.

Let's break down the sections:

Mailbox Async Processing State

mailboxid : 9fb6600b-0965-e811-a97f-000d4a161089

(GUID of the mailbox record)

hostid: Null         

(This will contain the name of the async server processing the request if mid-process)

processingstatecode : 0         

(This will be 1 if async is currently processing this mailbox and you see a hostid populated)

processinglastattemptedon : 12/3/2018 3:23:36 PM       

(This is the last time the mailbox attempted to process)

Mailbox Synchronization Methods

(EmailRouter here means Server-Side Sync)

incomingemaildeliverymethod : EmailRouter

outgoingemaildeliverymethod : EmailRouter

actdeliverymethod : EmailRouter

Mailbox Enabled State

(These states are tied to the test/enable being successful from a server-side sync perspective)

enabledforincomingemail : True

enabledforoutgoingemail : True

enabledforact : True

Mailbox Idle State

(Shows how many times the mailbox was processed with no Email items or Appointments, Contacts and Tasks to synchronize)

noemailcount : 3        

(This means 3 sync cycles ago, an email was promoted from the Exchange mailbox into Dynamics)

noactcount : 2      

(This means 2 sync cycles ago, an Appointment, Contact or Task was promoted from the Exchange mailbox into Dynamics)

Mailbox Backoff Parameters

(Shows when the mailbox is scheduled to process next. These are the fields I use the most)

postponemailboxprocessinguntil : 12/3/2018 3:28:37 PM

(This value controls when the Asynchronous Processing Service will run on this mailbox, which actually performs a synchronization)

postponesendinguntil : 11/29/2018 6:07:17 PM    

(If this value is the same or before the postponemailboxprocessinguntil value, asynchronous emails will be sent from Dynamics on the next sync)

receivingpostponeduntil : 12/3/2018 3:28:37 PM 

(If this value is the same or before the postponemailboxprocessinguntil value, emails in Exchange will attempt to sync with Dynamics)

receivingpostponeduntilforact : 12/3/2018 3:25:34 PM         

(If this value is the same or before the postponemailboxprocessinguntil value, Appointments, Contacts and Tasks will attempt to sync with Dynamics)

Mailbox Test and Enable Parameters

testemailconfigurationscheduled : False         

(This will be true if test/enable is pending)

testemailconfigurationretrycount : 0       

(This enumerates how many times the test/enable process has attempted to retry in the event of a failure)

testmailboxaccesscompletedon : 11/29/2018 6:07:22 PM         

(This provides the time the last test/enable was run on this mailbox)

postponetestemailconfigurationuntil : 11/29/2018 6:12:22 PM         

(If this value is the same or before the postponemailboxprocessinguntil value, test/enable will be run on the mailbox)

Mailbox Last Sync Cycle Information

lastsuccessfulsynccompletedon : 12/3/2018 3:23:37 PM         

(This provides the time that the mailbox last performed a sync without running into an exception/ error)

lastsyncerror: Null     

(This will provide limited exception details of what error occurred on the last mailbox sync attempt, if one occurred)

lastsyncerrorcode: Null         

(This will provide the exception code for the last error, if available)

lastsyncerrorcount : 0   

(This will enumerate how many consecutive times the same error has occurred for the mailbox synchronization)

lastsyncerroroccurredon : 10/17/2018 5:41:58 PM         

(This provides the time the last error occurred for the mailbox. If this time is before the lastsuccessfulsynccompletedon and processinglastattemptedon times, no error happened on the last sync)

itemsprocessedforlastsync : 2       

(This enumerates how many Exchange items were successfully promoted on the last sync cycle)

itemsfailedforlastsync : 0  

(This enumerates how many items succeeded the promotion criteria, but failed to promote to Dynamics)

Email Server Profile Details

(This section provides details on the Email Server Profile configured to this mailbox)

Email Server Profile General Settings (servertype: 1 is others, 0 is Exchange)

servertype : 0      

(0 is Exchange, 1 is others, such as POP3)

useautodiscover : True       

(False indicates that the EWS URL is explicitly defined in the Email Server Profile)

maxconcurrentconnections : 108       

(This value defines how many simultaneous connections to Exchange that this Email Server Profile can handle)

minpollingintervalinminutes : 0      

(This value determines how often Asynchronous processing is attempted for a mailbox in minutes. The minimum value is 5, so “0” in this context means that mailboxes sync every 5 minutes)

Email Server Profile Incoming Email Settings

incomingauthenticationprotocol : AutoDetect     

(This value defines what authentication method is being used, such as through an impersonation account or via mailbox credentials etc..)

incomingcredentialretrieval : S2S

(This defines the authentication credentials being used, such as S2S (server-to-server), OAuth, etc..)

incominguseimpersonation : False      

(This defines if an account with Application Impersonation is being used for incoming email synchronization)

incomingusessl : True   

(This defines if synchronization of incoming items uses certificates for encryption)

incomingportnumber : 443     

(This defines the port being used for incoming synchronization)

Email Server Profile Outgoing Email Settings

outgoingauthenticationprotocol : AutoDetect      

(This value defines what authentication method is being used, such as through an impersonation account, via mailbox credentials etc..)

outgoingcredentialretrieval : S2S         

(This defines the authentication credentials being used, such as S2S (server-to-server), OAuth, etc..)

outgoinguseimpersonation : False     

(This defines if an account with Application Impersonation is being used for outgoing email synchronization)

outgoingusessl : True      

(This defines if synchronization of outgoing items uses certificates for encryption)

outgoingportnumber : 443    

(This defines the port being used for outgoing synchronization)

Recent Trace Log details

(This section provides the last 10 mailbox specific logs for the associated mailbox. We will not be covering troubleshooting of the different errors here. Some of these will be more straightforward and some will require more telemetry review from a Microsoft resource. That is where the MailboxId and ActivityId can be used to correlate to additional logging.)

tracecode : 126

errortypedisplay : ExchangeSyncServerServiceError

errordetails : T:391

ActivityId: a2559263-94de-4886-bc28-5fc7511d4f5f

>Exception : Unhandled exception:

Exception type: System.Net.WebException

Message: The remote server returned an error: (503) Server Unavailable.   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)   at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.EndGetResponse(IAsyncResult asyncResult)   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.EndGetEwsHttpWebResponse(IEwsHttpWebRequest request, IAsyncResult asyncResult) -- End stack trace -- Exception type: Microsoft.Exchange.WebServices.Data.ServiceRequestException

Message: The request failed. The remote server returned an error: (503) Server Unavailable.   at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.EndGetEwsHttpWebResponse(IEwsHttpWebRequest request, IAsyncResult asyncResult)   at Microsoft.Exchange.WebServices.Data.SimpleServiceRequestBase.EndInternalExecute(IAsyncResult asyncResult)   at...

tracecode : 2

errortypedisplay : UnknownIncomingEmailIntegrationError

errordetails : ActivityId: 08657d6f-7e7b-424c-a974-6de3d4le2ae4a

>Error : ?<ResponseMessageType xmlns:q1="http://schemas.microsoft.com/exchange/services/2006/messages" p2:type="q1:FindItemResponseMessageType" ResponseClass="Error" xmlns:p2="http://www.w3.org/2001/XMLSchema-instance"><q1:MessageText>Mailbox move in progress. Try again later., Cannot open mailbox.</q1:MessageText><q1:ResponseCode>ErrorMailboxMoveInProgress</q1:ResponseCode><q1:DescriptiveLinkKey>0</q1:DescriptiveLinkKey></ResponseMessageType>

tracecode : 52

errortypedisplay : IncomingEmailServerServiceError

errordetails : ActivityId: b66413dd-c51e-43d1-9404-adb044abf655

>Error : System.Net.WebException: The request failed with HTTP status 503: Service Unavailable.

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)

at Microsoft.Crm.Asynchronous.EmailConnector.ExchangeServiceBinding.EndFindItem(IAsyncResult asyncResult)

at Microsoft.Crm.Asynchronous.EmailConnector.FindItemsStep.EndCall()

at Microsoft.Crm.Asynchronous.EmailConnector.ExchangeIncomingEmailProviderStep.EndOperation()

Thanks for reading!

Aaron Richards

In this blog series, we will explore building out DevOps processes and practices for Dynamics 365 Customer Engagement (CE) by utilizing YAML based Azure Pipelines. In this first blog, we will cover version controlling solutions and the basics of automating deployment.

What is DevOps?

DevOps is the union of people, process, and products to enable continuous delivery of value to our end users. The contraction of “Dev” and “Ops” refers to replacing siloed Development and Operations to create multidisciplinary teams that now work together with shared and efficient practices and tools. Essential DevOps practices include agile planning, continuous integration, continuous delivery, and monitoring of applications. (Sam Guckenheimer, What is DevOps?)

Getting started

If you already have a Dynamics 365 CE environment, Azure DevOps, you're familiar with the basics, and have a solution ready to go skip to Moving solutions between environments. If you are new to all of this, please continue from here.

The most basic workflow of Dynamics 365 CE development, deployment, and functional testing consists primarily of manual processes. We modify, export, and import a solution(s) into a downstream environment, then manually test to ensure there are no issues with solution layering, integrations, etc. That gets old quick, what we really want is to automate as much as we can. The first step we can take is to implement the practice of version control for our solutions and automate solution imports to various environments.

What we will need

• Azure Repo
  • If you are unfamiliar with Azure DevOps and need to set up a new account please start with the Get started guide for Azure Repos documentation, then come back to this blog.
• Dynamics 365 CE environment trial or follow along using your own environment.

Assumptions

• • Some experience with customizing Dynamics 365 CE

• A basic understanding of version control
  • There are a number of tools to make version control easier such as GitHub Desktop, Sourcetree, and GitKraken to name a few. For this series, it is expected that you know the basics or will learn Git for version control.

Version control for solutions

Solutions in Dynamics 365 CE are in essence a package containing any customization we've done to our environment that we can export from one environment then import into various environments. When exported from an environment, solutions are in the form of a zip file. When that zip file is unzipped, the output directory contains folders for plugins, web resources, and any workflows we have made as well as XML files defining the schema of any customization we have done. In the zipped format, our schema definition is contained in one massive file. Consider this zip file as a binary, or in layman's terms, a tidy package with a fancy bow, i.e. It may look nice but it's not easy to see what's inside and it's a poor format for version control.

Enter Solution Packager. Solution packager essentially takes our Dynamics 365 CE solution zip file and breaks it out into a logical folder structure by decomposing the contents. The resulting output shows a more granular view of our solution and is considerably more friendly for version control as you can see from the example screenshots below.

Example zip file directory structure

Example extracted solution directory structure

Note:

• The extra granularity of an unpacked solution makes it ideal for team development. Developers can pack and import a solution from a repository into their various development environments, make changes, export, unpack, and commit the solution back to the repository. For more on the benefits of using solution packager and example scenarios check out the article Use source control with solution files on Microsoft Docs.
• Solution Packager does not completely remove merge conflicts but does make it easier to manage them.

Download solution packager

To get started we will need to download a copy of the Dynamics 365 CE Core Tools Nuget package locally, this package includes Solution Packager.

Steps to download

1. Open a PowerShell Terminal
2. Navigate to the directory that you want the tools downloaded to.
   • Note that the code below will create the folder structure Tools\CoreTools in the current directory
3. Copy & paste the code below into the terminal or new script window if using PowerShell ISE and run the script.

    $sourceNugetExe = "https://dist.nuget.org/win-x86-commandline/latest/nuget.exe"
    $targetNugetExe = ".\nuget.exe"
    Remove-Item .\Tools -Force -Recurse -ErrorAction Ignore
    Invoke-WebRequest $sourceNugetExe -OutFile $targetNugetExe
    Set-Alias nuget $targetNugetExe -Scope Global -Verbose
    
    ##
    ##Download CoreTools
    ##
    ./nuget install  Microsoft.CrmSdk.CoreTools -O .\Tools
    md .\Tools\CoreTools
    $coreToolsFolder = Get-ChildItem ./Tools | Where-Object {$_.Name -match 'Microsoft.CrmSdk.CoreTools.'}
    move .\Tools\$coreToolsFolder\content\bin\coretools\*.* .\Tools\CoreTools
    Remove-Item .\Tools\$coreToolsFolder -Force -Recurse
    
    ##
    ##Remove NuGet.exe
    ##
    Remove-Item nuget.exe

*The code above was sourced from the Download tools from NuGet Microsoft Docs article.

Extract solution file

Now that we have Solution Packager downloaded we can work on adding our solution to version control. If you are following along with your own setup you will need to export a solution from your environment as both managed and unmanaged. It is recommended to use an unmanaged solution for development purposes and managed for all other environments. If you do not have a solution file handy grab a copy of our example unpacked solution from our GitHub repo in the Lesson-1 folder. Note that Azure DevOps build pipelines integrate with Azure Repos, GitHub, and GitHub Enterprise repository. In this example, I am using GitHub.

PowerShell command to extract Dynamics 365 CE solution:

The following PowerShell command allows us to extract both our managed and unmanaged solution files, this is accomplished by setting the packagetype to both and the zipfile value to the path to our unmanaged solution zip file. Solution packager will search the same directory for a solution file with the name containing the unmanaged solution name but ending in _managed.zip, for example, contoso.zip and contoso_managed.zip.

.\Tools\CoreTools\SolutionPackager.exe /action extract /packagetype both /zipfile <path to file>\contoso.zip /folder <path to output directory>

Once unpacked, submit the output to version control.

Moving solutions between environments using a YAML script

Now that we have an unpacked solution and it has been checked into version control we can work on automating deployment. I tend to follow the GitHub Flow development workflow that assumes anything in the master branch is always deployable. Following this pattern, we will build a simple YAML script that triggers on commits to the master branch and performs the steps outlined below. For a full explanation of YAML schema for Azure Pipelines check out aka.ms/yaml.

YAML script steps

• Create a new build pipeline
• Define name, trigger, and pool type
• Download and install Solution Packager
• Pack solution from repository
• Publish build artifacts
• Import solution into target environment

Create a new build pipeline

First and foremost we will need a new build pipeline, so let's head over to our Azure DevOps project repository to get this started.

1. Navigate to your Azure DevOps project repository. For example, https://dev.azure.com/{username}/D365-CE-DevOps-Tutorial
2. Click Pipelines, then click Builds.
3. Click New, then click New Build Pipeline
4. On the next screen, we will select the location of your unpacked solution: Azure Repos, GitHub, or GitHub Enterprise. Note that choosing GitHub requires that you authorize Azure DevOps to access the repository, the behavior otherwise is the same in our build pipelines.
5. Select the repository containing the unpacked solution files.
6. Configure pipeline using the Starter pipeline option.

*Note that choosing starter pipeline will name you file azure-pipelines.yml so you'll want to remain the file later and update the path in your pipeline settings.

Define name, trigger, and pool type

Now that a basic pipeline has been set up we can begin building out the steps needed to pack and deploy our solution file. For this example, we will be using the build pipelines only as YAML support for release pipelines is still under development at the time of writing. Please review the Publish build artifacts for more details.

We can start by deleting the contents of the starter template and adding our own definitions for the build name, trigger, and pool.

name: $(BuildDefinitionName)-$(Date:yyyyMMdd).$(Rev:.r)

trigger:
- master

pool:
  vmImage: 'vs2017-win2016'

Explanation of schema:

• Name - represents build number format. Default build numbers are whole numbers, we can do better than that.
• Trigger - specific branch to trigger on, if no branch is provided, commits to any branch will trigger continuous integration.
• Pool - essentially the type of OS and tools set you want to have your build run on. In this case, we are using Visual Studio 2017 on Windows Server 2016, but there are other available build agents as well.

Download and install Solution Packager

A build agent is provided to us as a blank workspace, so we will need to download and install the necessary tools to complete our build. We will first download and install Nuget so that we can install the Dynamics 365 CE Core Tools Nuget package. Solution Packager resides in the core tools package and is nested down a few levels. Rather than deal with a long string to point to SolutionPackager.exe, we will do some clean up to make it easier to reference the tool and to make this step easier to reuse in the future.

Append the following code the pipeline script, if you get a syntax error ensure Steps is not indented.

steps:
- script: md tools
  displayName: 'Create tools directory'

- powershell: |
    Invoke-WebRequest `
      -Uri https://dist.nuget.org/win-x86-commandline/latest/nuget.exe `
      -OutFile tools\\nuget.exe
  displayName: 'Download nuget.exe'

- powershell: |
    tools\\nuget.exe install Microsoft.CrmSdk.CoreTools -O tools
    md "tools\\CoreTools"
    $coreToolsFolder = Get-ChildItem tools | Where-Object {$_.Name -match 'Microsoft.CrmSdk.CoreTools.'}
    move "tools\\$coreToolsFolder\\content\\bin\\coretools\\*.*" "tools\\CoreTools"
    Remove-Item "tools\\$coreToolsFolder" -Force -Recurse
  displayName: 'Install CoreTools'

Note that script above is essentially the same as the script we used in the Download solution packager section.

Save and run to ensure your setup is correct and make adjustments as needed.

Pack Solution from repository

With Solution Packager installed we can turn our attention to packing up our extracted solution for importing into our target environment(s). To do that we will need to pack the solution back into its original zip file state using, this can be accomplished by reversing the extraction process we completed earlier in this blog.

- powershell: |
    Start-Process tools/CoreTools/SolutionPackager.exe `
    -ArgumentList `
      "/action: Pack", `
      "/zipfile: $(Build.ArtifactStagingDirectory)\packedSolution\$env:SolutionName.zip", `
      "/folder: $env:SolutionPath", `
      "/packagetype: Both" `
    -Wait `
    -NoNewWindow
  env:
    SolutionPath: $(solution.path)
    SolutionName: $(solution.name)
  displayName: 'Solution Packager: pack solution'

• /action - available options are pack and extract
• /zipfile - the location and desired name of packed solution zip file.
  • $(Build.ArtifactStagingDirectory) is a predefined build variable that points to a directory that is purged with each new build so no need for cleanup in our script.
• /folder - the location of our unpacked solution that will be packed.
• /packagetype - Defines whether the solution should be packed as managed or unmanaged. Both will create both unmanaged and managed solution zip files.

In the code above you will have noticed that there is now an environment variable section, env, that contains two variables: SolutionPath and SolutionName. These variables allow us to define the path to our extracted solution and desired packed zip file name. We will define the variables in our pipeline settings variables tab and enable the variables for modification at queue time. Note that the solution name that you provide is a friendly name, the actual unique name and localized name(s) that will be used in Dynamics are stored in the solution.xml file of the packed and unpacked solution.

Steps to create pipeline variables

For our script to be able to access the variables defined in our script, we will need to manually create them in our pipeline settings using the following steps:

*Don't forget to click Save and run before attempting to navigate away.

1. Click the ellipsis (…) on the top right of the page and in the drop-down click Pipeline settings, this will take you to the visual pipeline designer.
2. Click the Variables tab.
3. Click + Add in the pipeline variables view to add a new variable.
4. Perform step 3 twice and add the following variables
   • name: solution.name
     • value: <desired zip file name. In this example, contosoUniversity>
   • name: solution.path
     • value: <path to repository location of extracted solution. in this example, Lesson-1/ExtractedSolution/>
5. Checkmark both as Settable at queue time
6. Click the Save & queue drop down and click Save
7. Enter a comment and click Save

*If you queue your build ensure the branch where your YAML file is saved is correct or you may get an error. Or navigate back to the pipeline YAML editor and run from there.

Publish build artifacts

Next, we are going to publish our build artifacts even though release pipelines are not available for YAML builds at the time of writing this blog. In a future update, we will be moving our solution imports to a release pipeline and start breaking our script out into templates. For now, on build completion, you will be able to see and download the artifacts and the build will handle imports.

Check out the Pipeline artifacts in Azure Pipeline article to learn more about publishing and consuming build artifacts.

- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: $(Build.ArtifactStagingDirectory)
    artifactName: drop
  displayName: 'Publish build artifacts'

Save and run to ensure your setup is correct. again, ensure you don't have extra spaces as the script is space sensitive. You can click the build queue notification message to see the build steps run in real-time and view the execution history. Once the build is complete, you should see a new button labeled Artifacts on the top left of the page. Click this drop-down to view and download your solution file artifact(s). Clicking Release will automatically add build artifacts from this build to your release pipeline, for this blog we will not be doing this because YAML for release pipelines is not out yet.

Import solution into target environment

Finally! Let's deploy our solution to an environment.

The following snippet will download and install Microsoft.Xrm.Data.Powershell, then perform a synchronous import of our managed solution.

- powershell: Install-Module Microsoft.Xrm.Data.Powershell -Scope CurrentUser -Force
  displayName: 'Install Microsoft.Xrm.Data.PowerShell'

- powershell: |
    $connection = Get-CrmConnection `
      -ConnectionString `
        ("AuthType = Office365;" + `
        "Username = $env:ServiceAccountUpn;" + `
        "Password = $env:ServiceAccountPassword;" + `
        "Url = https://$env:EnvironmentName.crm.dynamics.com")

    Import-CrmSolution `
      -conn $connection `
      -SolutionFilePath $(Build.ArtifactStagingDirectory)\packedSolution\$($env:SolutionName)_managed.zip
  env:
    EnvironmentName: $(environment.name)
    SolutionName: $(solution.name)
    ServiceAccountUpn: $(serviceAccount.upn)
    ServiceAccountPassword: $(serviceAccount.password)
  displayName: 'Import solution'

Important - We have added some environment variables so we will need to edit our pipeline settings once more following the steps in the Steps to create pipeline variables section.

This time for step 4 we will add 3 new variables

• name: environment.name - value: <Dynamics 365 CE org name e.g. contoso.crm.dynamics.com, name only>
• name: serviceAccount.upn
  • value: example@contoso.onmicrosoft.com
• name: serviceAccount.password
  • value: hopefully not hunter2
  • For password variable be sure to select the lock symbol to change the type of the field to secret.

*Mark variables that you want to modify at queue time to settable at queue time.

That's it, now you can click Save and run on the top left of the editor. Once the build has been kicked off click the build name in the notification banner to watch the build run. After the build has successfully completed your solution will be imported into the specified target environment and that's that.

Stay tuned for more blogs that improve and build upon this example setup.

If you are interested in this topic and would like to do some further self-study I encourage you to check out the Solution Lifecycle Management: Dynamics 365 for Customer Engagement apps white paper.

*[CE]: Customer Engagement

References

• Example code - https://github.com/paulbreuler/D365-CE-DevOps-Tutorial