Debugging DLL Registration Issues

Did you know that the HKEY_CLASSES_ROOT registry hive (in Windows 2000 and above) is a figment of your imagination?

I was helping a coworker troubleshoot an application that runs as a service that accesses another homebrewed COM component. Of course, it worked on his machine, but not when deployed on a test machine. Running the normal Filemon/Regmon tag team, we traced it down to an issue instantiating the COM component. Regmon reported a registry key in HKEY_CLASSES_ROOT could not be found. Double clicking the line in Regmon (I really love that feature) opens up Regedit to the key that couldn't be found. I immediately guessed the root cause of the problem, and asked the developer the question above: Did you know that the HKEY_CLASSES_ROOT registry hive (in Windows 2000 and above) is a figment of your imagination?

He responded, "Huh? Its right there," as he pointed to the topmost hive in Regedit. A quick pointer to MSDN followed by browsing to HKEY_CURRENT_USER\Software\Classes and HKEY_LOCAL_MACHINE\Software\Classes showed the obvious - the key that Regmon showed as "not found" was really in the HKEY_CURRENT_USER section. As you may know, an application that runs as a service is run under the LocalSystem account (assuming no impersonation is done in the service, a user is not specified in the Service Control Manager, etc.). This account does not share the HKEY_CURRENT_USER hive with the user that installed the software, nor the currently logged in user, and was therefore unable to read the registry key necessary to instantiate the COM component since it did not exist for LocalSystem.

So what caused the problem? In an MSI created by most authoring tools, an option is given to the user to install the application for the "current user" or "all users". This ultimately sets the ALLUSERS property. In addition, the Windows Installer engine will add the registration information to either the HKCU or HKLM sections as defined by the ALLUSERS setting. The only documentation related to this behavior is buried in the SelfReg table, but not in the documentation for the property, tables, or actions that are truly affected by this. Bear in mind that for items in the SelfReg table, the DllRegisterServer() function is called in the context of the current user if ALLUSERS is set to a "this user only" setting, and "elevated" if it is a per-machine installation.

It is tempting to skip the bland and usually useless "User Information" dialog when creating your installations so that your installation doesn't get on people's nerves, but remember that while setting this property is important, you surely do not want to violate rule #20 while trying to appease rule #10.

Oh, and if you are interested in how I actually fixed this issue: since the per-user installation makes no sense for this application, and the "User Information" dialog was never shown, a custom action of type 51 came in handy to set ALLUSERS to 1 at the start of both the UI and InstallExecute sequences. Some ICE type validators may flag the installation if ALLUSERS is set in the property table. If you forcibly set ALLUSERS to 1, make sure that the LaunchCondition table checks that the user is Privileged and that Version9X is not set.