In the past few months, multiple Catapult customers reported an issue with Skype for Business client prompting end users to enter exchange hybrid credentials and repeat for the credentials.

The details below provide exactly what is observed, what are the causes and what are the fixes.

User Reported Issue:

End-users see the following on Skype for Business or Lync client. When the end user enters the credentials by clicking “Enter Credentials” button, end-user prompted again for the same – as if the credentials are either invalid or something goes wrong. The Calendar, conversation history etc. functionality of SFB client does not work for the end user.

Additionally, the end user sees an error/warning on bottom right corner and details with that error states that “EWS cannot connect to Exchange hybrid. It will keep trying in the background”.

Administrator Reported Issue:

The Skype for Business administrator see an event log with event ID 32054 – LS Storage Service:

Investigation:

The following commands were executed on both Skype for Business server and Exchange, and data collected to isolate the issue:

Skype for Business PowerShell command –

Test-CsExStorageConnectivity -SipUri <SIP URI of User> -Verbose

The output of above command shows that Autodiscover execution is happening on-prim and then online and failing. The command restarts the same execution multiple times before failing command with following error –

2018/07/25 16:44:28.786 Autodiscover, received GetUserSettings response, duration Ms=5836, response=<NULL>

Server stack trace:

   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

   at Microsoft.Rtc.Internal.Storage.IStorageService.EndExecuteCommand(IAsyncResult asyncResult)

   at Microsoft.Rtc.Management.Lyss.Cmdlets.LyssCmdletCommon.ExecuteExCommand(StoreOperation operation, String sipUri, BaseRequestType ewsRequest,

Nullable`1 autoCreateParentFolder, IStorageService& client, Boolean reAuthorize), inner exception=. Please check event log and trace for relevant

information.

At line:1 char:1

+ Test-CsExStorageConnectivity -SipUri <SIP URI of User> -Verbose

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (:) [Test-CsExStorageConnectivity], FaultException

    + FullyQualifiedErrorId : ErrorExecuteExchangeCommandFailedWithFaultException,Microsoft.Rtc.Management.Lyss.Cmdlets.TestExStorageConnectivityCmdlet

VERBOSE: Is command successful: False.

Test failed.

The verbose output shows that something is not working correctly on Exchange side.

Exchange PowerShell Commands –

The following commands were executed on Exchange PowerShell:

Get-ClientAccessServer | fl *Auto*

Get-MapiVirtualDirectory | FL server,*url*,*auth*

Get-WebServicesVirtualDirectory | FL server,*url*,*oauth*

Get-OABVirtualDirectory | FL server,*url*,*oauth*

Get-AutoDiscoverVirtualDirectory | FL server,*url*,*oauth*

The above commands showed different URLs of exchange CAS and mailbox servers. However, it should be pointed to specific hybrid server URL which supporting CAS role.

Additionally, executed the following –

The Autodiscover SRV record is missing from internal DNS.

The investigation showed multiple issues and reasons combine causing the actual issue.

Solution:

With help of my colleague, Jim Baker, I figured out the solution of this issue.

Please note that this issue is specific to Exchange environment where two or more Exchange versions deployed on-prem and this on-prem environment has a hybrid setup with Exchange Online (O365). The issue is seen with following on-prem Exchange version combinations:

• Exchange 2007, Exchange 2013 (Hybrid Server) and Exchange Online
• Exchange 2010, Exchange 2013 (Hybrid Server) and Exchange Online
• Exchange 2010, Exchange 2016 (Hybrid Server) and Exchange Online
• Exchange 2013, Exchange 2016 (Hybrid Server) and Exchange Online
• Exchange 2013 (Hybrid Server), Exchange 2016 and Exchange Online

Autodiscover SRV record missing –

Skype for Business or Lync client uses the Autodiscover SRV record to find the EWS location of the Exchange. When SRV record is missing, it just fails. Usually, in single Exchange on-prem environment, the Autodiscover A record or external Autodiscover allow Skype for Business or Lync client to find the EWS URL and client move forward. However, in the case of multiple Exchange versions, the SRV record is the only way Skype for Business or Lync client determines the EWS location.

The solution is to define AutoDiscover SRV record on internal DNS

• On the DNS server, click Start, click Control Panel, click Administrative Tools, and then click DNS.
• In the console tree for InCapital.com domain, expand Forward Lookup Zones, and then right-click the <User Domain>.com
• Click Other New Records.
• In Select, a resource record type, click Service Location (SRV) and then click Create Record.
• Click Service, and then type _autodiscover
• Click Protocol, and then type _tcp
• Click Port Number, and then type 443
• Click Host offering this service, and then type the Autodiscover.incapital.com.
• Click OK.
• Click Done.

Exchange PowerShell shows multiple Autodiscover URL –

The multiple URL returns multiple points for Skype for Business or Lync client to connect with using web services. However, web services protocol does not support DNS load balancing – this fails the connectivity with the URL and Skype for Business or Lync client fails to connect to Autodiscover to get correct EWS information.

The solution is to update the AutoDiscover and CAS Url and make it consistent. The PowerShell command that allows to set it correctly is:

Set-ClientAccessServer

Set-MapiVirtualDirectory

Set-WebServicesVirtualDirectory

Set-OABVirtualDirectory

Set-AutoDiscoverVirtualDirectory

Exchange Back End Certificate –

By default, Exchange back-end IIS service is assigned with a self-signed Exchange certificate. This causes the certificate trust to be broken between Skype for Business or Lync client and Exchange EWS when the client is sending credential to Exchange.

As shown above “Microsoft Exchange” is a self-signed certificate. This certificate does cause an issue for Skype for Business client connectivity.

The solution to change the self-signed certificate to a certificate that is used for Exchange as follows –

• Start IIS Manager, expand sites and select “Exchange Back End”
• Under Action Select Bindings…

• Select https with port 444 (as shown above) and click Edit…

• Drop Down on SSL Certificate and Select appropriate certificate (in the figure below Wildcard 2018 is exchange certificate)

• Click OK
• Click Close
• Run IISRESET on an Administrator CMD

Autodiscover Exchange Front End IIS Problem –

After peeling above three problems, the fourth one becomes clear. The root cause of the issue is that on Exchange Front End Autodiscover, there is no knowledge of multiple Exchange Autodiscover and how to reach to the exact Autodiscover for the end user when more then one on-prim exchange versions exists along with Exchange Online. The Exchange Front End Autodiscover logic fails – which is the main culprit of the repeat credential prompt etc.

Basically, The Autodiscover in IIS Exchange Back End service know about Hybrid and on-premise environment and has built-in logic with services. WSDL as well as supporting executables. However, The IIS Default Web Site (Exchange Front End) does not know about the on-premise environment OR exchange hybrid. When Autodiscover is requested from Exchange Front End server to Exchange back-end server, multiple entries are returned and Exchange front-end unable to handle it as it does not have any supporting logic available. This causes failure of data via the Autodiscover for EWS and other Exchange services. The following screenshots provide some details –

Default Web Site (Exchange Front End) –

AutoDiscover on Exchange Back End –

To resolve the above issue, the following steps are required –

1. Backup the files from “Default Web Site – AutoDiscover” using File Explorer located at the following location:
2. 
3. Copy the files from “Exchange Back End – AutoDiscover” using File Explorer located at the following location:
4. 
5. Paste all these files to “D:Program FilesMicrosoftExchange ServerV15FrontEndHttpProxyAutodiscover” folder using a file explorer
6. Run IISRESET on an Administrator CMD

This issue is not documented anywhere by Microsoft and specifically apply to multiple Exchange versions on-prim.