Needles in a Haystack: DisableLoopbackCheck – Failed Installation of OpsMgr 2012 SP1 Databases and Reporting with Server 2012 and SQL 2012 SP1 over Hyper-V and ESX 5.1 (E1000E vs. VMXNET3 Networking)

Recap of situation: Failed Installation of OpsMgr 2012 SP1 on Server 2012 and SQL 2012 SP1 over Hyper-V and ESX 5.1 (E1000E vs. VMXNET3 Networking)

This started becoming a real problem when I found the System Center Operations Manager 2012 SP1 Installation could not complete successfully in an all-in-one or remote DB\Reporting (SSRS)configuration.  The installation of SSRS natively shows a good health of the virtual directories for Reports and ReportServer. I was able to create a successful lab installation using Hyper-V, but could not perform a successful installation using ESX.

***Workarounds posted at the bottom of this page***

One specific error I ran into:

Setup is unable to create database on SQL Server instance <Alias or CNAME>. Please make sure that the current user has permissions to create database on the SQL Server instance specified.

What I verified: Using SQL Server Management Studio, I connected to the Instance using the CNAME and created a new database successfully.

Another issue I noted when installing Operations Manager Reporting:

Here’s some related links with similar issues although the MOM Team blog by JC Hornbeck has a slightly different error that is unrelated.

http://blogs.technet.com/b/momteam/archive/2012/10/24/kb-installation-or-upgrade-of-reporting-for-system-center-2012-operations-manager-fails-with-error-0xffffffff.aspx

http://social.technet.microsoft.com/Forums/en-US/operationsmanagerdeployment/thread/e5936efa-5f25-4d0f-9187-67469ac5ea90/

http://social.technet.microsoft.com/Forums/en-US/operationsmanagerdeployment/thread/e5936efa-5f25-4d0f-9187-67469ac5ea90/

http://support.microsoft.com/kb/2771907

http://thoughtsonopsmgr.blogspot.com/2013/01/om12-sp1-upgrading-sql-server-2008-sp1.html

http://thoughtsonopsmgr.blogspot.nl/2013/01/om12-sp1-sql-server-2012.html

Stages during the OpsMgr Reporting Installation:

     20.  Processing final configuration

NOTE: After this, you can perform a Reset of SSRS to its default configuration.  Utilize ResetSRS.exe from within the SupportTools folder of the System Center 2012 Operations Manager Media by running this within an Admistrator Command prompt:

Command = \SupportTools\ResetSRS.exe MSSQLSERVER

What is changed?

***
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Reporting]
"DefaultSDKServiceMachine"="mySCOMserver.mydomain.com"

***     

web.config:
***
      <dependentAssembly xmlns="urn:schemas-microsoft-com:asm.v1">
        <assemblyIdentity name="Microsoft.ReportingServices.ProcessingCore" publicKeyToken="89845dcd8080cc91" culture="neutral" />
        <bindingRedirect oldVersion="9.0.242.0" newVersion="10.0.0.0" />
      </dependentAssembly>
***
  <appSettings>
    <add key="ManagementGroupId" value="23b322bf-6965-70c9-3503-a0b44868dd47" />
  </appSettings>
***

rsreportserver.config
***
      <Extension Name="Windows" Type="Microsoft.EnterpriseManagement.Reporting.Security.GroupAuthorization, Microsoft.EnterpriseManagement.Reporting.Security">
        <Configuration>
          <Connection>
            <ServerName>server.domain.com</ServerName>
          </Connection>
        </Configuration>
      </Extension>
    </Security>
    <Authentication>
      <Extension Name="Windows" Type="Microsoft.EnterpriseManagement.Reporting.Security.GroupAuthentication, Microsoft.EnterpriseManagement.Reporting.Security">
        <Configuration>
          <Connection>
            <ServerName>server.domain.com</ServerName>
          </Connection>
        </Configuration>
      </Extension>

***
    <ReportItems>
      <ReportItem Name="EnterpriseManagementChartControl" Type="Dundas.ReportingServices.DundasChart,MicrosoftRSChart" />
      <ReportItem Name="DundasGaugeControl" Type="Dundas.ReportingServices.DundasGauge,DundasRSGauge" />
    </ReportItems>

 

I noticed that my Hyper-V lab machines all had a loopback adapter by default.  I had to perform the registry fix on my Hyper-V guests to DisableLoopbackCheck in order to complete the installation (details in ‘This results in secton’). With the loopback adapter fix in mind, I had a hunch and decided to search some VMWare forums that got some ideas flowing on that front.  This seemed to be two separate issues. 

What I found:

In ESX, you must use the E1000E NIC for OpsMgr 2012 SP1 using SQL 2012 SP1 on top of Windows Server 2012.  You cannot use the high performance NIC called VMXNET3, even though both have support for Server 2012.

Note: You MUST also disable the loopback check.  You will also see failures of SQL, SSRS and OpsMgr components when using CNAMEs without disabling the loopback adapter.  I also saw failures when not using CNAMEs and installing OpsMgr Reporting until we performed the workaround.

The root cause is related to:

This results in:

• Name resolution issues
• Unable to utilize CNAMEs
• Unable to create SQL Directories
• Unable to copy specific files during the installation process
• Unable to UNC using DB CNAME
• \ADMINSHARE$">\\<ALIAS>\ADMINSHARE$
• Able to UNC using Server name
• \ADMINSHARE$">\\<SERVER>\ADMINSHARE$
• Unable to install SQL RDBMS or SSRS
• Unable to Create OpsMgr databases during Install
• Incomplete Installation of OpsMgr RS with Dundas and Enterprise Management dll’s not being copied during the step ‘ Set OM Extensions’ and ‘Set Security Extensions”

Workaround performed on all virtual ESX guests:

1. If you are using ESX 5.1 for Server 2012 Support, change the ESX guest adapter to use the E1000E NIC
• This workaround from the KB on VMWare’s site does not work in this case, you must use E1000E: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004779

Workaround performed on all machines:

1. Open regedit.exe
2. Go to HKEY_LOCAL_MACHINE / SYSTEM / CurrentControlSet / Control / Lsa
3. Create REG_DWORD with name DisableLoopbackCheck and value 1
4. Disable the NIC
5. Re-enable the NIC
6. Try to UNC to the following paths and rerun the installation

• \\<ALIAS>\ADMINSHARE$
• \\<SERVER>\ADMINSHARE$
• Example: \\server01\c$ or cname of \\opsdb\c$

 

Referenced articles:

 

• Note: This article was not the resolution for loopback in VMWare, instead it was changing the NIC to E1000E and performing the DisableLoopbackCheck registry edit:
• http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004779

 

http://support.microsoft.com/?kbid=926642

http://social.technet.microsoft.com/Forums/en-US/winservergen/thread/8b99bef3-434a-41a7-bff4-a29ad004e0e7/

 

 

Method 2: Disable the authentication loopback check

Re-enable the behavior that exists in Windows Server 2003 by setting the DisableLoopbackCheck registry entry in the

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa registry

subkey to 1. To set the DisableLoopbackCheck registry entry to 1, follow these steps on the client computer:

1. Click Start, click Run, type regedit, and then click OK.

2. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

3. Right-click Lsa, point to New, and then click DWORD Value.

4. Type DisableLoopbackCheck, and then press ENTER.

5. Right-click DisableLoopbackCheck, and then click Modify.

6. In the Value data box, type 1, and then click OK.

7. Exit Registry Editor.

8. Restart the computer.

Note You must restart the server for this change to take effect. By default, loopback check functionality is turned on in Windows Server 2003 SP1, and the DisableLoopbackCheck registry entry is set to 0 (zero). The security is reduced when you disable the authentication loopback check, and you open the Windows Server 2003 server for man-in-the-middle (MITM) attacks on NTLM.