Category Archives: Exchange 2016

Email, Exchange 2016, Exchange 2019

Brightmail does not deliver email to Distribution group members

Leave a comment

The Story

Note: This article assumes you have Brightmail Gateway

When you try to send an email to a particular Exchange Distribution group Group@domain.com the result is either users don’t exist or you might get the following error if you test with Microsoft Test connectivity online tool.

Error:

The server returned status code 554 – Transaction failed. The server response was: 5.7.1 Delivery not authorized

Other related errors

‘554 5.7.1: You are not allowed to connect’

clip_image001

Cause:

Because the group has been cached in the Symantec gateway with its old members, The result could be an error that users don’t exist or delivery is not authorized.

Solution:

To solve this problem, You need to go to Brightmail gateway Administration > Directory Integration and click on your AD Directory > Advanced and hit on Clear Cache.

This would cleared the cached group and take the most recently updated group and its members.

This should resolve the problem.

clip_image002

How to clear the DDS cache in Messaging Gateway

https://knowledge.broadcom.com/external/article?legacyId=tech132131

Exchange 2016, Exchange 2019, Skype for Business 2015

Skype for Business IM integration with Exchange 2016 OWA–Part 2

Leave a comment

This article is a completion of Part 1, Click here to go to Part 1

Configuration Steps – Part 2

7. On Exchange: Enable OWA VD Instant Messaging
8. On Exchange: Enable Messaging on OWA Policy
9. On Exchange: Create Enterprise Application for Skype Pool.
10. On Exchange: Create new SettingOverride for Skype for Business.
11- Generate a new Certificate for Exchange IM
12. Assign the newly imported certificate to IIS Exchange Back End site
13. On Exchange: Restart the WebAppPool
14. Log out and sign back in to OWA to Check
15. Troubleshooting methods

    7- On Exchange Server: Enable OWA VD Instant Messaging

    Part of enabling IM integration between Exchange and SfB is to enable OWA Virtual Directory to allow this. The below cmdlet does the job for you on all your Exchange Servers

    From Exchange, Launch Exchange Management and run the following cmdlet

    Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -InstantMessagingEnabled $true -InstantMessagingType Ocs

    clip_image001[6]_thumb

    8- On Exchange: Enable Messaging on OWA Policy

    Run the following to enable Messaging for Owa Policy

    Get-OwaMailboxPolicy | Set-OwaMailboxPolicy -InstantMessagingEnabled $true -InstantMessagingType Ocs

    clip_image001[6]_thumb[1]

    9- On Exchange: Create Enterprise Application for Skype Pool.

      From Exchange Management shell Run the following cmdlet

      Cd $exscripts

      .\Configure-EnterprisePartnerApplication.ps1 -AuthMetadataUrl “https://sbg-pool01.domain.com/metadata/json/1” -ApplicationType Lync

      The AuthMetadataUrl is going to be your local Skype for Business Pool URL. This URL should work in your Exchange server without any Certificate error. Meaning that the certificate assigned to your Skype for Business pool should already be imported to Exchange Servers to trust this URL.

      image_thumb[14]

        If your previous configuration is correct then you should see the “The Configuration has Succeeded” Message.

          10- On Exchange: Create new SettingOverride for Skype for Business.

          Notes:

          • To configure the same settings on all Exchange 2016 and Exchange 2019 servers in the Active Directory forest, don’t use the Server parameter.

          New-SettingOverride -Name “<UniqueOverrideName>” -Component OwaServer -Section IMSettings -Parameters @(“IMServerName=<Skype server/pool  name>”,”IMCertificateThumbprint=<Certificate Thumbprint>”) -Reason “<DescriptiveReason>” [-Server <ServerName>]

          The Thumbprint you use here will define if whether IM will work or not as this what secures the communication between Exchange and Skype. If you use the wrong certificate your Integration will fail and users wont be able to login to IM through OWA.

          11- Generate a new Certificate for Exchange IM

          IMPORTANT NOTE:

          In order for IM in OWA to work the certificate you will generate must have its common name set as mail.domain.com to match the configuration.

          Using Digicert tool on Exchange Server I will generate the CSR of the new certificate

          Click on Create CSR

          image_thumb[15]

          Choose SSL certificate type and make sure you choose Mail.domain.com as CN

          In the SANs type all of the involved servers (Skype for Business Frontends, Mailbox servers in FQDN and in Hostnames as in the screenshot below). and click on Generate

          image_thumb[16]

          • Go to your CA Server’s CertSRV URL and copy the CSR code there to generate the new certificate.
          • Import the new certificate to the current server, then export it in PFX format and import it to all the Exchange Servers you’re planning to use.

          image_thumb[18]

          • After importing the certificate I will verify that I can see the private key

          image_thumb[19]

          Click on the Details and copy the Thumbprint or from MMC right click the certificate > Properties give it a friendly name e.g. (IM) and then from Exchange Management shell you can copy the Thumbprint directly.

          Get-ExchangeCertificate | select thumbprint,friendlyName

          image_thumb[20]

          Now use the previous script to create the setting Override for OwaServer.

          Things you can change are in bold “Name, IM Servername Value, and the Thumbprint value”.

          New-SettingOverride -Name “IM Override” -Component OwaServer -Section IMSettings -Parameters @(“IMServerName=SBG-Pool01.domain.com“,”IMCertificateThumbprint= 28E4B1BA0F2FCB1535AF199F02A64EFC78367F2D“) -Reason “Configure IM”

          image_thumb[21]

          If you enter the server parameter to use a single server you can change that by using. Note that you must not use FQDN but rather only the server’s hostname.

          Get-SettingOverride | Set-SettingOverride -Server sbg-mx01,sbg-mx02

          image_thumb[22]

          This should generate an event ID 112 on Exchange servers involved in the deployment.

          clip_image001[9]_thumb

            12. Assign the newly imported certificate to IIS Exchange Back End site

            Once the certificate is in the server store, You will be able to easily find in from IIS and bind it to the Exchange Back End site.

            This is the most crucial step to get IM to work in OWA. Don’t worry about breaking up Exchange Sites or Powershell. If you have added Exchange Servers Hostnames and FQDNs in this certificate then you should be good.

            • Now Launch IIS
            • Click on Exchange Back End
            • Select Binding
            • Click on the 444 port and edit
            • Select the newly generated certificate that has the mail.domain.com as CN. (This certificate must also have all Exchange Servers hostnames and FQDNs set as SANs)

            image_thumb[23]

            image_thumb[24]

            Make sure you change the backend cert to the new on all the involved Exchange Servers.

            13. On Exchange: Restart the WebAppPool

            Restart-WebAppPool MSExchangeOWAAppPool

            image_thumb[25]

              14. Log out and sign back in to OWA to Check

              Log out of OWA and back in and check if you are able to Login to IM . It should normally sign you in automatically but in case of an error then you should see it.

              image_thumb[29]

              In case of an error you should see the following.

              image_thumb[27]

              If it works then you should see the presence

              image_thumb[28]

              15. Troubleshooting Methods

              If you follow the above steps correctly then it should work especially when applying the right certificate for your Exchange Back End IIS part however if you face an error then you should do the following steps to troubleshoot the error

              • Set the Eventlog for Instant Messaging on Exchange from Low to High

              Set-EventLogLevel -Identity “sbg-mx01\MSExchange OWA\InstantMessage” -Level High

              image_thumb[30]

              • Look in the following path for errors

              C:\Program Files\Microsoft\Exchange Server\V15\Logging\OWA\InstantMessaging

              • Check the Healthset of the OWA Instant Messaging.

              Get-ServerHealth -HealthSet OWA.Protocol.Dep -Server sbg-mx01 | Format-Table Name, AlertValue –Auto

              image_thumb[31]

              Get-MonitoringItemIdentity -Server sbg-mx01 -Identity OWA.Protocol.Dep | Format-Table Identity,ItemType,Name -Auto

                image_thumb[32]

                Ref

                https://docs.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/configure-im-integration-with-owa?view=exchserver-2019

                https://docs.microsoft.com/en-us/exchange/high-availability/managed-availability/health-sets?view=exchserver-2019

                Exchange 2016, Exchange 2019, Skype for Business 2015

                Skype for Business IM integration with Exchange 2016 OWA–Part 1

                Leave a comment

                The Story

                A good and detailed documentation is everything we need to implement any kind of project especially if it’s an integration between two different servers that perform different roles.

                And with PKI involved the complications multiply thus a good article write up is what we need.

                Previously I have tried a test lab with Skype for Business 2015/2019 IM Integration with Exchange 2016/2019 and the result was a complete failure and endless search for what’s missing to get IM to work from OWA?

                image

                ERROR

                Upon completion of the steps mentioned in Microsoft’s Official documentation and after restarting Exchange IIS or OWAAppPool you will see this when you try to login to OWA with your user

                There’s a problem with instant messaging. Please try again later.

                image

                MS Official Documentation

                In their Official documentation Microsoft says that the certificate in question must be trusted by all the servers involved meaning Skype for Business Frontend and Mailbox Servers.

                Meanwhile this is true, it still would not get the IM to login/work although it might drop the initialize event ID 112 in the event log.

                clip_image001

                Here is what MS says about the certificate.

                Exchange and Skype for Business integration requires server certificates that are trusted by all of the servers involved. The procedures in this topic assume that you already have the required certificates. For more information, see Plan to integrate Skype for Business Server 2015 and Exchange. The required IM certificate thumbprint refers to the Exchange Server certificate assigned to the IIS service.

                REF URL: https://docs.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/configure-im-integration-with-owa?view=exchserver-2019#what-do-you-need-to-know-before-you-begin

                image

                Step by Step Deployment

                To do things the way that should get this to work, I will detail steps one by one so we can be sure to get the positive results we are all waiting for when dealing with Exchange and Skype for Business.

                Exchange IM URL 1: mail.domain.com

                Skype for Business Pool FQDN: SBG-Pool01.domain.com

                Autodiscover URL : Autodiscover.Domain.com

                Prerequisites

                1. For Default and Web Service Internal, Your Skype for Business Frontend Server/Pool must use a certificate that is generated from an internal CA which you can use later to generate Exchange’s IM Certificate.
                2. UCMA must be installed (Doesn’t matter if version 4 or 5) both are supposed to work with Exchange 2016.
                3. Local Certification Authority must already be deployed in the domain.

                Configuration Steps – Part 1

                1. On SfB: Set CsAuthConfiguration Autodiscover URL for Skype server to find Exchange Autodiscover
                2. On SfB: Get-CsSite to see what is the current site ID.
                3. On Exchange: Check AutodiscoverServiceInternalURI
                4. On SfB: Create new Partner
                5. On SfB: Create new Trusted Application Pool
                6. On SfB: Create new Trusted Application ID

                Configuration Steps – Part 2

                7. On Exchange: Enable OWA VD Instant Messaging
                8. On Exchange: Enable Messaging on OWA Policy
                9. On Exchange: Create Enterprise Application for Skype Pool.
                10. On Exchange: Create new SettingOverride for Skype for Business.
                11- Generate a new Certificate for Exchange IM
                12. Assign the newly imported certificate to IIS Exchange Back End site
                13. On Exchange: Restart the WebAppPool
                14. Log out and sign back in to OWA to Check
                15. Troubleshooting methods

                Prerequisites

                1- Update or Create Server Default and Web Service Internal Certificate for SfB Pool servers

                The certificate installed on the Skype for Business Pool Frontend servers must be generated from a local Certification Authority which can be trusted by Exchange Server services.

                The Certificate generated for Skype for Business pool as in the below screenshot is generated from my CA and includes the names of the servers:

                • Skype for Business Pool
                • Skype for Business Frontend FQDNs
                • Exchange Servers
                • Autodiscover FQDN
                • Lyncdiscover.domains.com
                • Lyncdiscoverinternal.domains.com
                • sip.domains.com
                • meet.domains.com
                • dialin.domain.com
                • External.domain.com

                image

                image

                2- UCMA must be installed

                On both Exchange and Skype for Business servers I already have UCMA 4.0 version installed, but if you don’t have it or have an older version then you can’t continue without it.

                image

                3- Make sure you have a Local Certification Authority deployed in your domain.

                Configuration Steps – Part 1

                1- On SfB: Set CsAuthConfiguration Autodiscover URL for Skype server to find Exchange Autodiscover

                For Skype for Business Server to find Exchange Autodiscover Service point and to be able to authenticate servers we’ll be using the below cmdlet

                This enables both servers to authenticate and share information when needed and without user’s interference.

                Set-CsOauthConfiguration -ExchangeAutodiscoverUrl https://autodiscover.domain.com/autodiscover/autodiscover.svc

                image

                image

                Ref:

                https://docs.microsoft.com/en-us/powershell/module/skype/set-csoauthconfiguration?view=skype-ps

                2- On SfB: Get-CsSite to see what is the current site ID.

                Getting a site ID will be useful for later use to setup the Trusted Application Pool.

                On Skype for Business Management shell. Type the following

                Get-CsSite

                So the Site ID is 1. I will keep this for later use

                image

                3- On Exchange: Check AutodiscoverServiceInternalURI

                Specify the AutodiscoverServiceInternalURI for internal Autodiscover service. Make sure it points to your public URL and certificate not the internal one otherwise your users will get a certificate error through Outlook and might cause IM chat not to work.

                The Cmdlet would be

                Get-ClientAccessService | Set-ClientAccessService –AutoDiscoverServiceInternalUri https://autodiscover.domain.com/autodiscover/autodiscover.xml

                image

                4- On SfB: Create new Partner Application

                On Skype for Business Server, Launch Management Shell and use this cmdlet to add Exchange as a trusted Application to the SfB topology.

                New-CsPartnerApplication -Identity Exchange -ApplicationTrustLevel Full -MetadataUrl “https://autodiscover.domain.com/autodiscover/metadata/json/1“

                image

                5- On SfB: Create new Trusted Application Pool

                New-CsTrustedApplicationPool -Identity mail.domain.com -Registrar sbg-pool01.domain.com -Site 1 -RequiresReplication $false

                image

                6- On SfB: Create new Trusted Application ID

                From SfB Management Shell run the following cmdlet .

                New-CsTrustedApplication -ApplicationId OutlookWebAccess -TrustedApplicationPoolFqdn mail.domain.com -Port 5199

                image

                Finally

                clip_image001[4]

                ADMT, Cross Forest, Exchange 2010, Exchange 2016

                an Exchange mailbox was mistakenly migrated over another user’s object used by another user

                Leave a comment

                The Story

                If you ever used Prepare moverequest command to migrate a user and forgot to use ADMT to rewrite user’s properties with the old attributes. You might have used ADMT again to rewrite the properties.

                If you use ADMT you will need to exclude all Exchange Attributes from the source since its already copied using Prepare-move request script however, in some cases some people do make mistakes and you might have came through the same mistake my colleague  have done during one of these extremely complicated Cross forest Migrations where you’d prepare a CSV files through PowerShell and names wouldn’t match Sam accounts.

                Don’t Panic

                If however, you forgot again to exclude the Exchange attributes while using ADMT then you most likely wont see the user in the Target forest which will cause to panic thinking the user is gone .. But no the user is not gone don’t panic.

                When you look for the user’s mailbox on the target forest after the move request is completed you’ll get an error reporting the user can’t be found

                image

                Solution

                To fix the problem you’ll need to change to attributes only for this migrated user. (In the target forest after user mailbox move is completed).

                The attributes are

                msExchRecipientDisplayType    1073741824
                msExchRecipientTypeDetails    128

                The wrong Attributes are as following.

                image

                You will need to fix them to look like the following

                image

                Once you apply the change you’ll need to wait for a minute or few depending on your AD replication speed.
                The problem will be then solved

                image

                Exchange 2010, Exchange 2013, Exchange 2016, Exchange 2019, Security

                Microsoft Exchange Vulnerability affects all Exchange versions

                Leave a comment

                image

                CVE-2020-0688 | Microsoft Exchange Validation Key Remote Code Execution Vulnerability

                Security Vulnerability

                Date of Publishing: February/11/2020

                Microsoft has announced a vulnerability has been found in all Exchange Server 2010 through 2019 versions, The vulnerability allows an attack to send a specially crafted request to the affected server in order to exploit it.

                When could this happen?

                A remote code execution vulnerability exists in Microsoft Exchange Server when the server fails to properly create unique keys at install time.

                Knowledge of a the validation key allows an authenticated user with a mailbox to pass arbitrary objects to be deserialized by the web application, which runs as SYSTEM.

                The security update addresses the vulnerability by correcting how Microsoft Exchange creates the keys during install.

                Affected Versions:

                • Microsoft Exchange Server 2010 Service Pack 3 Update Rollup 30
                • Microsoft Exchange Server 2013 Cumulative Update 23   
                • Microsoft Exchange Server 2016 Cumulative Update 14   
                • Microsoft Exchange Server 2016 Cumulative Update 15   
                • Microsoft Exchange Server 2019 Cumulative Update 3   
                • Microsoft Exchange Server 2019 Cumulative Update 4

                image

                Solution:

                Until now Microsoft has not provided any solution or work around to cover this vulnerability.

                Mitigations

                Microsoft has not identified any mitigating factors for this vulnerability.

                Workarounds

                Microsoft has not identified any workarounds for this vulnerability.

                NOTE:

                Keep an eye on the below link for any change

                https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0688

                Active Directory, Exchange 2016, Exchange 2019, Exchange Online, Microsoft, Office 365, Powershell

                How to Sync Cloud User to On-premises AD ?

                Leave a comment

                The Story:

                I have got this client who constantly keeps on making the mistake of create user from Cloud and provision them with a license in an Exchange Hybrid environment.

                Although this is not difficult to fix but it’s not the recommended approach when creating a new user especially in a Hybrid environment since Exchange on-premises won’t recognize this user and most likely will consider any incoming emails from it as spoof or spam.

                How to Create a Cloud user from Exchange On-premises?

                From Exchange on-premises ECP Admin panel you have the option to directly create user on-cloud which will also create a user object on on-premises AD.

                image

                Second option – Using Powershell

                It’s not that much different than the Web UI option but it’s just for people who prefer using PowerShell than GUI

                Enable-RemoteMailbox –Identity User –RemoteRoutingAddress user@yourTenant.mail.onmicrosoft.com

                The reason to follow those two methods is due to the need of Exchange on-premises being aware of each of those users so mail flow between Exchange on-premises and Online would not get affected and route this users mail to the wrong place or flag it as spammed or spoof …etc.

                The Real Question now is: How to Sync Cloud User to On-premises AD ?

                If by mistake we created a user on Cloud (Office 365) and we forgot to create an AD User for this account, that user might already have started using his account on Office 365 (Sharepoint, Exchange, Teams) etc.

                There also might be the intention of moving users from Cloud to On-premises Exchange in case the company wanted to decrease their spending on cloud users and in this case when Migrating a cloud user to on-premises you will get the following errors:

                image

                test3@domain.com

                Status: Failed

                test3@domain.com Skipped item details

                User status

                Data migrated:

                Migration rate:

                Last successful sync date:

                Error: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’. –> Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’.

                So how do we overcome this situation since syncing a user might require you to delete the cloud user and recreate it on AD?

                Solution:

                To sync the user from the Cloud to on-premises you will need to follow these steps :

                1- Create an on-premises Mailbox where the following attributes would be matching the cloud user

                • UserPrincipalname
                • ProxyAddresses
                • SamAccountName
                • Alias

                2- The Location of the OU where the On-premises user is going to be created must be provisioned by ADConnect (Azure AD Connect)

                You can look which of these OU are provisioned by Starting AD Connect Sync Manager

                image

                By verifying the user you created in the AD is in the right OU, You can now start AD Sync from PowerShell to speed up the process.

                image

                Below, You can see the user has been successfully synchronized to the cloud without any issue.

                image

                Now we’ll see it from the portal to confirm the user is synced with AD

                image

                Depending on the Source anchor being used in ADConnect there might be a GUID conflict or not, You will get an error similar to when trying to migrate the user in the beginning however you can solve this by replacing the cloud user’s GUID (ImmutableID) with the on-premises user which will force the user to merge with the On-prem user.

                Let’s confirm in our case if the user on-cloud has a matching GUID with the one on-premises.

                From CMD or Powershell you can use the following command to get the user’s ImmutableID (ObjectGUID) .

                ldifde -f c:\Test.txt -d “cn=Test3,DC=Domain,DC=com”

                image

                Checking the notepad we just exported you can see the Immutable ID on AD for the User test3 is IkTni9mw7Ee4YefeGpz7IA==

                image

                To be able to see the user on Office 365, We need to logon to MSOL through Exchange Online powershell

                Connect to Exchange Online’s powershell using your Online ECP.

                image

                Once you click on Configure this should download an executable file that will launch PowerShell Online which allows you to use the Modern Authentication (MFA) to use PowerShell safely.

                image

                Connect-Msoluser will connect you to Office 365 and you’ll be able to get the user’s properties and see if the Immutable ID is matching to the user’s GUID.

                Once you’re connect you can use the following cmdlet to get the user’s properties.

                Get-MsolUser -UserPrincipalName test3@domain.com |fl DisplayName,ImmutableID

                image

                You can see they are matching each other, In case there’s a conflict then you can simply set the online user’s Immutable ID to the on-premises user and that should solve the problem.

                Ref:

                https://support.microsoft.com/en-us/help/2956029/migrationpermanentexception-cannot-find-a-recipient-that-has-mailbox-g

                https://docs.microsoft.com/en-us/exchange/hybrid-deployment/create-cloud-based-archive

                Exchange 2016, Exchange 2019

                Move Request on Exchange 2019 (During failover) will warn you that it postponed due the move of DB

                Leave a comment

                The Issue:

                So while working on a new Exchange Migration project, I have encountered a weird issue where I could see users migration batch status complaining about being stalled due to (Big Funnel).

                The error is showing as in the below screenshot and it doesn’t occur instantly after you start the migration of the user but right after it starts.

                StalledDueToTarget_BigFunnel 68.47 MB (71,795,512 bytes) 20

                User StalledDueToTarget_BigFunnel 37.2 MB (39,003,538 bytes) 20

                User2 StalledDueToTarget_BigFunnel 14.71 MB (15,421,154 bytes) 20

                User3 StalledDueToTarget_BigFunnel 44.2 MB (46,345,009 bytes) 20

                User4 StalledDueToTarget_BigFunnel 4.647 MB (4,872,404 bytes) 20

                User5 StalledDueToTarget_BigFunnel 14.47 MB (15,169,768 bytes) 20

                User6  StalledDueToTarget_BigFunnel 171 MB (179,280,335 bytes) 20

                User7 StalledDueToTarget_BigFunnel 753.4 MB (789,980,880 bytes) 20

                User8 StalledDueToTarget_BigFunnel 18.35 MB (19,236,680 bytes) 20

                User9 StalledDueToTarget_BigFunnel 205.9 MB (215,951,208 bytes) 20

                User10 StalledDueToTarget_BigFunnel 166.2 MB (174,243,238 bytes) 20

                User11 StalledDueToTarget_BigFunnel 13.81 MB (14,481,739 bytes) 20

                User12 StalledDueToTarget_BigFunnel

                image

                Error Message

                Request ‘domain.com/CompanyUSER/Region1/User1’ (b5dbf3ff-21a1-4ec1-a29c-15b794a17386) failed.

                Error code: -2146233088

                Connection to the Content Transformation Service has failed.

                Context:

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                Operation: IMapiFxProxy.ProcessRequest

                OperationSide: Target

                b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMailbox.ExportMessages

                Operation: IMailbox.ExportMessages

                OperationSide: Source

                b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

                Flags: SkipItemValidation

                PropTags: (null)

                ——–

                >>>> Scheduled WorkItems: EnumerateFolderMessages(P:29792,R:0,S:0,C:14); EnumerateFolderMessages(P:29807,R:0,S:0,C:24,Cnt=3); WriteFolderMessages(P:0,R:0,S:0,C:686); EnumerateFolderMessages(P:30554,R:0,S:2,C:55); EnumerateFolderMessages(P:30612,R:0,S:0,C:36,Cnt=2); WriteFolderMessages(P:3,R:0,S:0,C:301); EnumerateFolderMessages(P:30975,R:0,S:1,C:21); WriteFolderMessages(P:2,R:0,S:0,C:97); EnumerateFolderMessages(P:31094,R:0,S:0,C:18,Cnt=6); EnumerateFolderMessages(P:31279,R:0,S:0,C:19)

                ————–

                The Microsoft Exchange Mailbox Replication service was unable to save changes to request.

                Request: ‘9a444721-80e2-4cf8-8c81-8a3afe3dc775’ (bbc2c66e-857e-4ba6-8462-9d66da73d400)

                Database: DB01

                Error:

                The request has been temporarily postponed because a database has failed over. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request when capacity becomes available on the new server hosting the database.

                image

                Looking at the event ID number 1114 it mentions there seems to be an issue with the request seems there might be an issue with the mailbox being moved.

                To dig deeper I am going to search some of the users reporting the same error by using their GUID

                image

                The property “DisplayName” with value “User LastName” is invalid. The value can’t contain leading or trailing whitespace.

                To resolve the problem, I am going to get the user’s mailbox and remove the trailing space in the end of the display name. You can safely use the below Powershell script to solve this problem however, if you don’t trust yourself or you’re not familiar much with Powershell, You can try it on a lab or a single test user for instance.

                Solution: (For a single user)

                Get-Mailbox -Identity USER | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

                image

                Solution: (For all users)

                Get-Mailbox | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

                clip_image001

                Some relevant errors you might encounter as you’re moving users to Exchange 2019

                Error code: -2146233088

                Connection to the Content Transformation Service has failed.

                Context:

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                Operation: IMapiFxProxy.ProcessRequest

                OperationSide: Target

                eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMailbox.ExportMessages

                Operation: IMailbox.ExportMessages

                OperationSide: Source

                eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

                Flags: SkipItemValidation

                PropTags: (null)

                ——–

                >>>> Scheduled WorkItems: EnumerateFolderMessages(P:14014,R:0,S:0,C:13); EnumerateFolderMessages(P:14029,R:0,S:0,C:15,Cnt=2); WriteFolderMessages(P:1,R:0,S:0,C:132); EnumerateFolderMessages(P:14192,R:0,S:0,C:17); WriteFolderMessages(P:1,R:0,S:0,C:48); EnumerateFolderMessages(P:14259,R:0,S:0,C:12,Cnt=4); EnumerateFolderMessages(P:14320,R:0,S:1,C:15); EnumerateFolderMessages(P:14337,R:0,S:0,C:20); WriteFolderMessages(P:2,R:0,S:0,C:126); EnumerateFolderMessages(P:14485,R:0,S:0,C:30)

                Exchange 2016, Exchange 2019, Microsoft Exchange, Office365

                Search and Delete certain Items/Folders from a Mailbox

                Leave a comment

                The Story

                During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:

                Error:

                Error: MigrationPermanentException: Mailbox dumpster size 50.87 GB (54,620,074,576 bytes) exceeds target quota 30 GB –> Mailbox dumpster size 50.87 GB exceeds target quota.

                image

                After some research it turned out that you can clean the dumpster using search-mailbox PowerShell cmdlet, Sync the user’s object with ADConnect and then continue the migration from the last failure.

                To solve the issue, Go on your Exchange on-premises and launch Exchange Management shell

                Solution:

                First, Let’s see the user’s dumpster and recoverable items

                Get-MailboxFolderStatistics -Identity “User” -FolderScope RecoverableItems | Format-Table Name,FolderPath,ItemsInFolder,FolderAndSubfolderSize

                image

                To Delete the dumpster only use this

                Delete dumpster only

                Search-mailbox -identity User -SearchDumpsterOnly –DeleteContent

                To delete a certain email with certain subject in the dumpster use the following:

                Get-mailbox “user”| search-mailbox –searchquery “Subject:’*'” –DeleteContent –SearchDumpsterOnly

                image

                The cmdlet will search and delete

                clip_image001

                image

                Reference:

                https://docs.microsoft.com/en-us/microsoft-365/compliance/search-for-and-delete-messagesadmin-help

                Exchange 2013, Exchange 2016, Exchange 2019, Microsoft, Security

                Microsoft Exchange 2013–2016-2019 Security Vulnerability (URGENT)

                Leave a comment

                CVE-2019-0586 | Microsoft Exchange Memory Corruption Vulnerability

                image

                Yesterday Microsoft has published an KB article about a security threat to Exchange 2013-2016 and 2019.

                The article mentions CU10-CU11 for Exchange 2016, CU21 for Exchange 2013 and RTM version of Exchange 2019 as well.

                I spoke to Microsoft of whether this issue affects earlier CU versions for Exchange and they confirmed it

                 

                image

                Vulnerability details:

                A remote code execution vulnerability exists in Microsoft Exchange software when the software fails to properly handle objects in memory. An attacker who successfully exploited the vulnerability could run arbitrary code in the context of the System user. An attacker could then install programs; view, change, or delete data; or create new accounts.

                Exploitation of the vulnerability requires that a specially crafted email be sent to a vulnerable Exchange server.

                The security update addresses the vulnerability by correcting how Microsoft Exchange handles objects in memory.

                 

                Download link:

                https://support.microsoft.com/en-us/help/4471389/description-of-the-security-update-for-microsoft-exchange-server-2019

                 

                Active Directory, Certification Authority, Email, Exchange 2010, Exchange 2013, Exchange 2016

                Create 10 years Certificate template for Skype for Business, Exchange Server .etc

                Leave a comment

                Having a certificate template that would last for years is a requirements that’s become more of a need during these days due to the amount of time it takes to renew the servers which are certificate dependent.

                Some servers would automatically be renewed their Certificate using GPO’s auto enrollment however when this doesn’t work or if you don’t like dealing with GPO and its headache then the best way to solve this is to create a template that would last for sometime and leave you at peace.

                Note that having a long term certificate (10 years for example) is not a good practice since encryption algorithm changes over the time and security related issues rise up every now and then so if you decided to go with this article and create 10 years template for your servers, you will need to keep an eye on latest news related to certificates, encryption and signature algorithm so they are not exploitable.

                To begin with this article, I will tell a small story of a company that had suffered production disasters due to this might look tiny kind of problem.

                A company called AP have deployed Lync 2013 (Currently Skype for Business) and decided to use the default CA (Webserver) template which lasts for 2 years by default.

                This company called me when their Lync servers were all down and PSTN calls were not going through so the first thing that came to my mind is to check Services and as soon as I saw services were not able to run I checked the eventlog’s Lync tab.

                The errors were mostly referring to an expired certificate. upon renewing the certificates for all servers everything went back to normal but that took long downtime and delayed the company’s productivity.

                Here came the idea of creating a long template which would last for 10 years, achieving this on Exchange is done through the following steps:


                On CA Server Find and open the Certification Authority MMC

                Right click on Certificate Templates and click Manage

                clip_image001[4]

                IN certificate templates console right click on Web Server and click Duplicate template

                clip_image002[4]

                Select Windows Server 2003 Enterprise

                clip_image003[4]

                clip_image004[4]

                Enable “Allow private key to be exported”

                clip_image005[4]

                clip_image006[4]

                Select Enroll for the Authenticated users

                clip_image007[4]

                Back to the Certificate Authority Console, Right click on Certificate Templates and click New -> Certificate template to issue and add the certificate template you created to the list.

                clip_image008[4]

                Web Server V2 is on top

                clip_image009[4]

                Let’s check it on Certserv IIS

                clip_image010[4]

                Certificate is generated for 5 years. The reason why the certificate is generated for 5 years is because the Certification Authority server’s Certificate is limited to 5 years.

                So the CA certificate must be set to longer then the client’s requested Certificate limit.

                clip_image011[4]

                Certification Authority Issuing Certificate validity period extending

                To change the Validity Period for the Root CA you can configure a CAPolicy.inf.  To create a CAPolicy.inf file that changes the lifietime of the certificate to 30 years, you would type the following into a text file, and save it with the name CAPolicy.inf in the

                C:\Windows directory,:

                [Version]

                Signature= “$Windows NT$”

                [Certsrv_Server]

                RenewalValidityPeriod=Years

                RenewalValidityPeriodUnits=30

                From <http://blogs.technet.com/b/xdot509/archive/2013/06/06/operating-a-windows-pki-renewing-ca-certificates.aspx>

                After this you will need to renew the CA certificate from the CA console : right click on your certification authority and choose All Tasks -> then choose -> Renew CA Certificate

                clip_image012[4]

                When you click on Renew CA certificate you will get the following prompt asking you to stop the CA to renew its Certificate, Click Yes

                clip_image013[4]

                Once you click on Yes the service will stop and you will get this window telling you if you would like to generate new public and private key it’s up to you to use a new or not but if you choose Yes the clients using the old Certificate might be provoked and you might need to install the new CA Certificate on all clients using GPO.

                Click Ok

                clip_image014[4]

                After clicking OK you will see that you were able to generate the new CA Certificate and then you can issue clients certificates.

                clip_image015[4]

                In order to allow the CA to issue certificates that has longer than the default age (2 years) you must run the following command line on the CMD on the CA server.

                clip_image016[4]

                Run the certutil commands above to change the maximum lifetime of certs issued by the CA

                certutil -setreg ca\validityperiodunits 30
                certutil -setreg ca\validityperiod years
                net stop certsvc net stop certsvc

                Now when you try to generate a new certificate for your Exchange or any other app you should choose the new template which is 30 years valid.

                clip_image018[4]

                clip_image019[4]