Hello,
Preface this post by saying a man from Exchange support said “This is the most complicated Exchange environment I’ve ever seen”.
That said this issue is pretty common and hopefully this post will help someone else.
We have an Exchange 2010 to 365 hybrid environment that look a bit like this:
We had an issue where users on our 365 tenancy couldn’t see on the Exchange 2010 on-premises free/busy info for users in Group2.contoso.com.
Now I know what your thinking, just compare the settings on Group1 to Group2, well due to company rules and politics I can’t….I can only troubleshoot group2 and the servers there.
So first things first, check users permissions and setup a test user and find the error in Outlook:
“No Information. No free/busy information could be retrieved” The recipient’s server could not be contacted. Contact your administrator.
Thanks to Babunski and his post I found this really good troubleshooting guide and everything looked ok:
- Firewall is fine,
- Network is ok,
- DNS surprising is working,
- Check Exchange online tool:
https://www.testexchangeconnectivity.com
- 365 to prem relationship is ok:
OrganizationRelationship -Identity "Exchange Online to On Premises Organization Relationship" | FL
- IIS Logs look ok, %SystemDrive%\inetpub\logs\LogFiles
- EWS logs look ok, %SystemDrive%\inetpub\logs\LogFiles
- Checked the external URL – seems ok.
Get-WebServicesVirtualDirectory | FL Name,Server,externalURL
- Check the IIS permissions with – this looked ok
Get-WebServicesVirtualDirectory | fl name,server,externalURL,ExternalAuthenticationMethods
- Checked IIS EWS and Autodiscover:
- Checked more relationship stuff – all ok
Get-OrganizationReationship -Identity "On Premises to Exchange Online Organization Relationship”
Next – contact support! 🙁
Before I contact support I did find one more URL that suggests to check the certs and import the cert you used to setup the federation onto the CAS server which unfortunately didn’t work for us:
Soooo here I am, time to contact support.
The first thing they checked was the local url on the client access server:
https://ClientAccessGroup2Server1.group.contoso.com/ews/exchange.asmx
So there’s an issue, basically we didn’t add the server to our wild card cert. So added the server names as Subject alternative names and imported it using PowerShell onto both Client access servers and then rebooted:
Enable-ExchangeCertificate -Thumbprint 011111111111111111111111111111111111 -Services SMTP,IIS
Fixed:
Checked the URLS set in Exchange:
Get-WebServicesVirtualDirectory | FL
Our internal URL was actually set to Client Access array for Contoso rather than group2.contoso.com so we changed this:
set-WebServicesVirtualDirectory –identity ClientAccessGroup2Server1 –internalurl https://CASARRAY.Group2.contoso.com/EWS/exchange.asmx
And rebooted again.
Next we disabled and re-enabled ISS security (this broke OOF for a while, we had to run this twice):
Set-WebServicesVirtualDirectory -Identity 'ClientAccessGroup2Server1\ews (Default Web Site)' -BasicAuthentication $false -WindowsAuthentication $false -WSSecurityAuthentication $false Set-WebServicesVirtualDirectory -Identity ' ClientAccessGroup2Server1\ews (Default Web Site)' -BasicAuthentication $true -WindowsAuthentication $true -WSSecurityAuthentication $true
So here are stuck…..
MS ran some traces using Extra:
And went away for a while and came back with:
Internet facing Site Conotso.com is able to look up the user and send a request to Group2 servers.
Testy1@contoso.mail.onmicrosoft.com: Request for Testy2@group2.contoso.com is being proxied to https://CASARRAY.Group2.contoso.com/ews/exchange.asmx Testy1@contoso.mail.onmicrosoft.com: Setting exception to all queries: Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: System.InvalidOperationException: Client found response content type of '', but expected 'text/xml'.
The request failed with an empty response.
at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling(). The request information is ProxyWebRequest type = CrossSite, url = https://ClientAccessGroup2Server1.group2.contoso.com:443/ews/exchange.asmxMailbox list = <Bert Test>SMTP: Testy2@group2.contoso.com, Parameters: windowStart = 30/07/2018 00:00:00, windowEnd = 10/09/2018 00:00:00, MergedFBInterval = 30, RequestedView = MergedOnly. ---> System.InvalidOperationException: Client found response content type of '', but expected 'text/xml'.
On group2 server we notice below error,
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/en-GB/library/System.ServiceModel.Diagnostics.EventLog.aspx</TraceIdentifier><Description>Wrote to the EventLog.</Description><AppDomain>/LM/W3SVC/1/ROOT/EWS-1-131788827699531225</AppDomain><ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/DictionaryTraceRecord"><CategoryID.Name>WebHost</CategoryID.Name><CategoryID.Value>5</CategoryID.Value><InstanceID.Name>WebHostFailedToProcessRequest</InstanceID.Name><InstanceID.Value>3221356547</InstanceID.Value><Value0>System.ServiceModel.ServiceHostingEnvironment+HostingManager/39086322</Value0><Value1>System.ServiceModel.ServiceActivationException: The service '/EWS/Exchange.asmx' cannot be activated due to an exception during compilation. The exception message is: This collection already contains an address with scheme http. There can be at most one address per scheme in this collection.
So the long and short of it is they think IIS is broken. The traffic is being passed to the Group2 services but these services are not passing the information back up the stream.
MS decided they wanted swap out the EWS web.config with a new one from:
c:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\exchweb\EWS
The reason being in the config file it was referencing:
<assemblyIdentity name="Microsoft.Exchange.Common.IL" publicKeyToken="31bf3856ad364e35" culture="neutral" /> <codeBase version="0.0.0.0" href="file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Common.IL.dll"/>
And it should be referencing (or where ever you install of Exchange is):
<assemblyIdentity name="Microsoft.Exchange.Common.IL" publicKeyToken="31bf3856ad364e35" culture="neutral" /> <codeBase version="0.0.0.0" href="file:///C:\Program Files\Microsoft\Exchange Server\V14\bin\Microsoft.Exchange.Common.IL.dll"/>
And another reboot.
Next we checked the logging from Outlook:
Which dumps files too: %Temp%\outlook logging
And they found this error:
Exception Type Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException Response Code ErrorProxyRequestProcessingFailed
This prompted MS to check the IIS bindings which were wrong
So we added some missing bindings using these command:
C:\Windows\system32>cd inetsrv appcmd set site /site.name:"Default Web Site" /+bindings.[protocol='net.tcp',bindingInformation='808:*'] SITE object "Default Web Site" changed appcmd set site /site.name:"Default Web Site" /+bindings.[protocol='net.pipe',bindingInformation='*'] SITE object "Default Web Site" changed appcmd set site /site.name:"Default Web Site" /+bindings.[protocol='net.msmq',bindingInformation='localhost'] SITE object "Default Web Site" changed appcmd set site /site.name:"Default Web Site" /+bindings.[protocol='msmq.formatname',bindingInformation='localhost'] SITE object "Default Web Site" changed
No change and still the same error: Response Code ErrorProxyRequestProcessingFailed
So we checked windows services and would you believe it but these dot net services were not installed:
net.tcp lisener adapter
net.pipe listener adapter
So we installed the missing features:
Rebooted both.
And Boom we are working!!!!!!!!!!!!!!!!!!!!!
