Exchange / Outlook - There is a problem with the proxy server's security certificate

Issue

After replacing a wildcard SSL certificate (*.domain.com) with a basic single-server certificate (webmail.domain.com) remote clients using Microsoft Outlook can no longer connect to their email accounts on an Exchange server using HTTP Proxy Method. Outlook displays below error and then repeatedly prompts for a password:

There is a problem with the proxy server's security certificate.
The name on the security certificate is invalid or does not match the name of the target site webmail.domain.com.
Outlook is unable to connect to the proxy Server. (Error Code 0)

Resolution

If remote clients use Outlook 2010 or Outlook 2013 they can manually change Exchange Proxy Settings in Outlook:

• Open MS Outlook and go to: Account Settings > More Settings > Connection > Exchange Proxy Settings
• Under "Only connect to proxy servers that have this principal name in their certificate:" replace "msstd:*.domain.com" with "msstd:webmail.domain.com"
  

Microsoft Outlook 2016 exclusively uses Exchange auto-discover to configure Exchange accounts so manually changing Exchange Proxy Settings in Outlook is not an option. Fist of all you need to ensure that MS Exchange auto-discover pushes correct certificate principal name:

• Login to the Exchange server and start Exchange Management Shell
• Run following command:
  Get-OutlookProvider -Identity EXPR | fl
• Check values:
  • CertPrincipalName - should have correct SSL certificate common/principal name in format: msstd:webmail.domain.com
  • Server - should be blank
• If CertPrincipalName is incorrect, run following command to modify it:
  set-OutlookProvider -id EXPR -Server "[servername]" -CertPrincipalName "msstd:webmail.domain.com"
• if Server is not blank run following command to clear it:
  set-OutlookProvider -id EXPR -server $null

Even when Exchange auto-discover is configured correctly, Outlook 2016 won't update an existing account with new certificate principal name. In my experience the best way to fix this is to dismiss all errors, then open Outlook and try to add the same Exchange account again. Outlook won't add it as a new account, but instead update the existing one with correct settings.

Source: https://www.mysysadmintips.com/windows/servers/772-exchange-outlook-there-is-a-problem-with-the-proxy-server-s-security-certificate