Tuesday, November 15, 2005

Custom Action Tutorial Part III – What we did in Part II

This is the third in a series of articles about building Custom Actions. If you are coming in late to the party, check out Part I and Part II first. If I tried to explain what we were doing and why in the last installment of this tutorial, you would be even more lost now than you probably are. This article is an attempt to explain exactly what it was we did in Part II and why. Let’s start by going over each project in our solution.

The HelloWorld project

We needed something to install. This happens to be it. I could have picked notepad.exe or something, but it is nicer to see how Visual Studio could grab project outputs and put them into an MSI fairly easily.

The CustomAction project

This one is really the meat of what we are trying to learn. The other projects are sort of extraneous if you are using a tool other than Visual Studio to build your MSI. Microsoft did a pretty good job of explaining why you may want to use Custom Actions in the first place, so I’m not going to rehash that.

We created a Custom Action DLL. The reason I chose a DLL is because it is the most flexible means of tightly integrating a Custom Action with the installation engine. Additionally, there is not much you cannot do in a Win32 C++ DLL. An executable Custom Action cannot interact with the MSI engine. A VBScript or JScript action does have this capability, but if you are not careful, these are not bulletproof, and usually won't allow you to do much anyway.

We added a "Module Definition File" (.def) file to the project. This .def file is read by the linker and is used (for our purposes, anyway) to prevent mangling of function names (to be technically correct, specify the calling convention to be used). Mangling is entirely normal when a DLL is created without using the extern "c" function declaration – in fact, it is called by the more pleasant name "Decorating" the function names. If you run "Dependency Walker" (depends.exe in the Bin folder of the Platform SDK), and open a DLL in it, the listing of function exports appears. Pick FrameDyD.Dll from the same Bin directory and look at it. You will notice some ? and @ symbols around the names of functions. The .def file allows us to not create these mangled exports – you can verify this by opening the custom action dll we built in our project and see this for yourself. Ultimately, this makes the process less prone to error when entering data in the custom action table of the MSI. When adding the Custom Action via Visual Studio, it will handle the mangling automatically, so for this specific project this was not strictly necessary. If you are adding this DLL using another installation designer, you will thank me for not mangling the function names. Anyway, the .def file must contain the function names of any function you wish to make callable from outside the DLL. In our case, we wanted all four of the functions we added to be callable by the MSI engine, which is why they were added to the DEF file. There are tons of other ways to accomplish this goal

The function signatures we added to the CustomAction.cpp file are based on the requirements of the DLL Custom Action type. We return an unsigned integer (UINT) and accept a single parameter that is the handle to the installation (MSIHANDLE). We must use this handle to interact with the MSI Engine. The UINT return type indicates our status to the MSI engine.

The file "msi.lib" is the link library for the MSI engine. All custom action DLL’s must link to this file in order to call any MSI functions. The instructions in Part I explained an alternate way to inform the linker what to link with. One way is not necessarily better or worse than the other. The functions defined in the header files msi.h and msiquery.h (that we included in the stdafx.h file) all require linking to this library.

We made additional solution configurations to give us a UNICODE project type. While this is not intended to be a primer on UNICODE, all NT kernel OS’s are UNICODE under the hood. Windows 95 through ME are not UNICODE, although they can support some UNICODE extensions with the Unicode Library. For dll's and executables that only will run under Windows NT, 2000, XP, or 2003, the only option you should use is UNICODE. Therefore, if you are targeting only NT based systems, build and use the UNICODE projects. If you are supporting Windows 9x based machines use the non Unicode versions. To make the Unicode transition easier, Microsoft provided the tchar.h header that we include in the stdafx.h file that deals with strings based on compiler settings (actually preprocessor macros), so you may write code that will compile and work as native UNICODE or ANSI based on compiler settings. The TEXT() macro used in the "Hello MSI!" exercise is an example of one such tchar.h macro. Future Custom Action examples presented here will use the tchar.h macros when the compatibility of the Custom Action spans both Windows 9x and NT based platforms. Actions targeting Windows NT based systems will only compile in UNICODE, and will not use the tchar.h macros.

The final thing we added to the CustomAction project was the header include for strsafe.h. This header file describes the safe string functions intended to replace the C/C++ standard library string functions as well as specific Windows string handling implementations. These functions always start with the word "String" followed by the type of count provided as a parameter to the function, either the "Cb" which is a count of bytes, or "Cch" which is a count of characters. Remember that Unicode implementations use 2 bytes per character as opposed to the ANSI 1 byte per character – so the Cb and Cch designations are important. The last important fact about this header is by default it will deprecate the unsafe string handling functions – if you use sprintf(), you will get a compiler warning.

The Setup1 Project

Hopefully we are pretty clear on the basics for the first part of this one – an EXE is generated, Visual Studio generates a basic MSI and adds entries in the Feature, Component, File, Shortcut, and other tables to install the HelloWorld exe and the shortcut for it.

When we added the Custom Actions, we did a few things that affected the compiled MSI. Use Orca (which you installed in Part I) to view the generated MSI and follow along - try to remember back to the Custom Action theory presented in Part I:

  1. Added a new Component and File table entry that installs the CustomAction.dll file.
  2. Added four entries to the CustomAction table, referencing the DLL that we installed with the product. Notice the Source column of the CustomAction table is a reference to the key of File table for our Custom Action DLL. The CustomAction table's Target column is the DLL entry point (remember this is nothing more magical than the function names in our .def file). The action name is just a randomly generated GUID. The type column starts its life as a Type 17 which indicates msidbCustomActionTypeDll + msidbCustomActionTypeSourceFile, and to this number is added the following:
    Action Type Decimal ValueDescription
    Uninstall 1041= 17 + msidbCustomActionTypeInScript
    Install 1041 = 17 + msidbCustomActionTypeInScript
    Rollback 1297 = 17 + msidbCustomActionTypeInScript + msidbCustomActionTypeRollback
    Commit 1553 = 17 + msidbCustomActionTypeInScript + msidbCustomActionTypeCommit
  3. Looking at the InstallExecuteSequence table, note that all but the uninstall custom action is scheduled before the RegisterUser Standard action, and the uninstall custom action is scheduled before the UnpublishComponents action. Note that these actions are conditionalized on the action state of the component, denoted by the dollar-sign - you can see this clearly in the InstallExecuteSequence table. In a nutshell, this condition will cause the custom actions to run only when the component it is tied to (the Custom Action DLL) is being added or removed from the system.
Visual Studio does not give us a bunch of options when dealing with Custom Actions, so we are pretty much stuck with what they offer – which is the ability to add an additional condition (other than the component install state), stuff what goes into the CustomActionData property fairly easy, and sequence the custom actions relative to each other. You cannot change sequencing of the custom action (like making it occur relative to another standard action other than RegisterUser or UnpublishComponents, embedding the custom action in the binary table, or add immediate custom actions. Other authoring tools offer the ability to be tons more flexible, but we are stuck with these limitations for now.

That concludes Part III of the Custom Action Tutorial. In the next part, we will learn how to debug the Custom Action.

1 comment:

Arjun said...

Hi Steven,

Thanks for such a wonderful tutorial. I am actually facing the following problem. How do I pass user set property data to the custom action dll? I am trying to use the MSIGetProperty in my DLL and it always returns am empty string?