Monday, May 4, 2015

Loss of OWA and ECP access "Something Went Wrong"

We lost access to OWA and ECP. users would get the "Something Went Wrong" message. After much troubleshooting, Microsoft were able to determine that the issue was with corruption to the "Canary Data" which is an AD attribute that is created during the first exchange 2013 schema preparation.

It creates 4 attributes while schema preparation or it may be even just one attribute (still getting to the bottom of whether there is always 4, after the following process we only had 1 but was fixed. I will update later).

msExchCanaryData0
msExchCanaryData1
msExchCanaryData2
msExchCanaryData3

This is a secret token that exchanges between the clients and the server for services OWA,ECP and other exchange web services. These values gets stored in the cookie collection of the clients browser. So for any OWA, ECP, EWS requests from clients, the browser sends the GUID value that is stored in the cache and compares it with the GUID that is in the URL (server). If they dont match then the request from the client is considered as malicious and blocked. Microsoft informed us that corruption of this data is rife in Exchange 2013 CU3 (which is what we are on (at the time of writing).

The solution is to delete that data and reboot every CAS and every mailbox server which causes the data to be rebuilt.


Right click the 'CN=Client Access' container and select properties. The find the msExchCanaryData entries, double click them and clear out the data so that they then have the value of "not set". NOTE: that is not the same as deleting the keys!

Then have fun rebooting everything, the keys should rebuild and you should be fixed.

We need to apply a more recent CU

Cheers


3 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. My AD 2012 R2 schema with Exchange 2013 extensions does not have msExchCanaryData3 so don't kick yourself looking for it. It just has msExchCanaryData0, msExchCanaryData1, and msExchCanaryData2.

    mliben@hotmail.com

    ReplyDelete