We’ve all been there – doing maintenance late at night. You think you’re just about done, and then something unexpected happens. Tonight was one of those nights =)
Rebooted the OCS Front End server and then the services would not start. The system event log recorded this error message:
“The Office Communications Server Front-End service terminated with service-specific error 3286774273 (0xC3E83201).”
After searching around, I came across only a handful of blogs that referenced a bug with the April 2009 update rolling back if the MSMQ service was not installed. That was not my situation – I had the MSMQ installed, and my server had been running fine for several months.
The trick to finding out more detailed information about this error was to drill down into the Office Communication Server event log.
I noticed Event ID 32014 – local certificate not found exception. Interesting….
The application threw an exception while starting.
The application Microsoft.Rtc.Applications.Cas threw the following exception when starting: Exception: System.Runtime.Serialization.SerializationException
> Message: The constructor to deserialize an object of type ‘Microsoft.Rtc.Internal.Sip.LocalCertificateNotFoundException’ was not found.
> TargetSite: Void CallStartAsync()
> StackTrace: at Microsoft.Rtc.ApplicationServerCore.ApplicationLoader.CallStartAsync()
> Source: Microsoft.Rtc.ApplicationServerCore
Why would I be missing a certificate? Checking the local certificate store showed valid certificates. Then I went back into the OCS MMC and saw that the expiration date was 9/27/2010. Shouldn’t be a problem, today is 9/9/2010 – I’ve still got time left on the cert, so that’s not the issue… or is it? I noticed that the only certificate on the machine had an expiration date of 8/16/2011 – next year. Somebody did me a favor and renewed my certificate for me – 30 days in advance. Who would do such a kind thing? I knew I had one entity to thanks – automatic certificate renewal, courtesy of Active Directory Certificate Services. This was by design. Ouch!
Here is my theory of what happened. On 8/16/2010 – a little more than one month before the local computer certificate was about to expire, the computer automatically requested an updated certificate from the certificate authority. The OCS services continued to run fine, because it apparently had cached the old certificate into memory. This scenario created a time bomb situation for the unsuspecting administrator. Once I restarted the OCS server, the services could not startup because the certificate that they were looking for was not there. The fix was easy – just right click on the server and assign the new certificate. After that, the services started right up!
I hope this helps someone searching on this same error message. It just seemed so wrong that this situation would happen because I still had time left on my certificate. I guess sometimes we don’t want the underlying Certificate Authority to be so courteous as to automatically renew certificates one month in advance.
The irony is that my System Center Operations Manager 2007 R2 had already alerted me that the certificate was going to expire soon – but I knew I had time, or so I thought!