Custom Action Tutorial Part I – Custom Action Types and Sequences

This is the first part of a multi-part series on Custom Actions in the MSI world. The articles are designed to be read in order, as each one will build from knowledge gained in the previous ones. To give you a short roadmap, the series will start with some dry and boring theory, and the subsequent articles will start us down a path of writing Custom Actions in unmanaged C++, C#, and script.

Custom Actions are used when the power of the Windows Installer engine's Standard Actions are not enough to accomplish a given installation task. For instance, installing or controlling a service is trivial using MSI Standard Actions, but creating a virtual directory in IIS is not. To create that virtual directory, you will need to write a custom action. Other examples of installation tasks that require Custom Actions can be found here.

Before beginning, I'd like to make sure you have the proper environment to work under. I'm going to use Microsoft's Visual Studio 2003 as the environment of choice for this series. The writing level is intended to be a walkthrough for someone who may be academically familiar with C/C++. I am going to handwave much of the Win32 enumerated types and definitions - the reader who is completely unfamiliar with these topics can find additional resources in books, the MSDN library, or around the web.

This part covers some necessary background on the different Custom Action types and Custom Action scheduling. Mostly this is dry and boring theoretical stuff, but essential to understand. Hopefully, I summarized it fairly well and won't cause anyone to drift off to sleep during their lunch hour.

Setting up your environment

For this first part, you only need the Platform SDK and a free MSI editor called Orca (distributed with the Platform SDK) installed. Future parts will require the following setup, so prepare yourself by setting this up. The development environment should be set up as follows: Install Visual Studio 2003, the current MSDN Library, and the most current Platform SDK. The Platform SDK contains new and updated header files and link libraries to support the latest OS platforms, components, and service packs. The installation of the Platform SDK installs a shortcut Microsoft Platform SDK in the Program Files menu. Open this and find the option to register the platform SDK directories with Visual Studio and run it. Finally, in the installation directory for the SDK, there is a directory "bin" and inside this you will find orca.msi. I recommend installing Orca, as my instructions are based on this. You may also substitute the MSI editor of your choice, just be aware of the differences.

Let's get started

We are going to use Orca to investigate the Orca.msi file. So, find the Orca.msi file, right-click on it, and select Edit With Orca, and play along.

The Two "categories" of Custom Actions

There are only two major categories of custom actions. The category of a custom action is indicated by a number in the type column of the CustomAction table (You should be finding this table in Orca as you read along). This number is actually a bitmask - for folks new to programming, if you looked at the number in binary form, the presence of a 1 or 0 in a particular location is how all the flags and options are interpreted by the MSI engine. I will give the hex (or Hexidecimal) representations of these flags throughout (all hex digits begin with a 0x to indicate they are hex). You can use the Windows "calc" program in the scientific view to enter the hex and convert it to binary to see which flag toggles which bit. Using hex (or even binary) is perhaps the best way to reverse engineer the decimal value from the table into the flags it represents. The two major categories of custom actions are:

• Deferred - This is a modifier of (in hex) 0x400 or 0x4000 in the Type column of the CustomAction table, which corresponds to what MSDN calls (and you do not need to understand what these are yet) msidbCustomActionTypeInScript or msidbCustomActionTypeTSAware, respectively. What is important is Deferred actions are actions that modify the state of the system in some way. There are also some notable restrictions on deferred actions - you cannot change the value of a property, you can only read the value of a single property, and you cannot interact with the MSI tables. These can only be sequenced between the InstallInitialize and InstallFinalize actions in the Sequence table, lest you encounter the dreaded 2762 error "Cannot write script record. Transaction not started." error. You will find out the reasons behind this error later in this article. Additionally, there are options to specify a special kind of deferred action of "Rollback" or "Commit." These will also be covered later in this article.
• Immediate - This is a modifier of zero in the Type column of the CustomAction table - in other words it is any entry in the Type column that does not contain the bitmasks described in the Deferred section above. Immediate actions are used for UI tasks, evaluating the state of the system, or preparing data or the system itself for later modification from a Deferred action. Immediate actions should never modify the state of the system.

The Orca.msi I am looking at has two different custom action types - 0x23 and 0x33. Neither of these numbers has the 0x400 or 0x4000 bits set, so this must mean they are Immediate Actions.

The Type column of the CustomAction table

Custom Actions have a Type - this is the base number (or bitmask) that gets entered in the Type column. This Type column tells the MSI engine both what they do and how to interpret the Source and Target columns of the CustomAction table. The major Types are:

Action	Hex Code	Description / Notes
Call a DLL function	0x01, 0x11	The DLL function that is called must be specifically written for the MSI engine.
Run an EXE	0x02, 0x12, 0x22, 0x32	Any EXE will run. By default, if the EXE returns a code other than zero, the action will fail. There is a flag available to turn off this behavior.
Abort the Installation	0x13	Aborts installation and displays a Formatted message from the target column.
Set a directory	0x23	Uses formatted text from the Target column to set a Directory.
Set a property	0x33	Uses formatted text from the Target column to set a Property.
Run some Jscript	0x05,0x15,0x25,0x35	Bear in mind the normally present WScript object is not available in these scripts.
Run some VBScript	0x06,0x16,0x26,0x36	Bear in mind the normally present WScript object is not available in these scripts.
Run a nested installation	0x07, 0x17, 0x27	Nested installations are evil. Don't do it. Use a setup launcher or AppSearch to enforce setup order.

Hopefully you see the pattern here - all the EXE actions contain 0x2, all the JScript ones contain a 0x5, etc.

The Source Column of the CustomAction Table

The second hex digit from the right in the CustomAction Type column tells us how to interpret the Source column. This is a bit more of a stretch then above, but you should see the trend:

Hex Code	Description / Notes
0x00	The Source column is a key into the Binary Table, where the the file needed to run this action is stored.
0x10	The file needed for this action is installed as part of this installation, the Source column is the key into the File table. This fact places restrictions on where this type of action can be placed in the installation sequence. In the case of the nested installation, the Source column points to the msi path inside this msi's source tree. For the abort action, the source column is ignored.
0x20	Source column is the key into the Directory table for setting the directory or indicating the EXE's working directory,Source column is the ProductCode for the nested installations,Source is null for the VBScript and Jscript actions. Note that you cannot return anything other than a success code with actions of this type. Actual script is stored in the Target column of the CustomAction table.
0x30	Source is the Property name where the value of the property will be set by the set property action, Source column contains the script code for script actions, or for the EXE action the Source column is the property that contains the path of the EXE to run.

The Target column of the CustomAction table

The Target column is dependent on the major type of the CustomAction, reading the MSDN Summary List of All Custom Action Types will explain this column in more appropriate detail. Looking back to the Orca.msi file - the actions listed translate to the following:

• 0x23 - Set the directory listed in the Source column to the Formatted Text in the Target column.
• 0x33 - Set the property listed in the Source column to the Formatted Text in the Target column.

How and When is a Custom Action run?
Now that we know a Custom Action is either Immediate or Deferred, and can be or do just about anything, we need to answer the question "How does a Custom Action get fired off?" Entering a Custom Action into the CustomAction table really only makes it available to be called - so just adding a line in the CustomAction table effectively does nothing. There are three methods of firing off a Custom Action:

• In the sequence tables. This is probably the most common way of running a custom action, and the focal point of our discussion for the remainder of this article series.
  
• In the ControlEvent tables. Essentially this is the method to launch a Custom Action in response to a UI event such as a button click. On Windows Server 2003 or better, custom actions run this way cannot send messages with MsiProcessMessage() calls, however Session.Message() calls from the automation interface work just fine (technically, prior to Windows Server 2003 neither message API officially works). Obviously, the installation must be running in full UI mode to fire these off. The Custom Action type launched by this method must be an Immediate type - remember we should never alter system state in the UI phase. I'm not going to cover how to run an Immediate Action from a dialog here using ControlEvent, perhaps in a later article.
  
• By a call to Session.DoAction() or MsiDoAction(). This is a way of calling one Custom Action from another. Although this won't make sense until it is explained later on, you can not call a Deferred Custom Action in this manner unless it is called while the "installation script" is being written - but it should be OK to call it from a custom action scheduled between InstallInitialize and InstallFinalize. More on the "installation script" concept later.

The Sequence tables

Before we can discuss the primary way Custom Actions are launched, you need a better understanding on how MSI sequences work. We will get into this in more detail later in this article so I am handwaving some important details here for the sake of getting a basic understanding. The typical installation is kicked off by double-clicking the MSI. This causes the installation engine to begin processing the InstallUISequence table (The processing of this table is skipped entirely if the UI mode is set to "basic" or "no UI" - both Full and Reduced UI modes actually do process this table). There are other sequences such as the AdminUISequence and AdvtUISequence that all follow the same logic for administrative and advertised installations. Hopefully you still have Orca open, so look at the tables as I describe them and follow along - sort the table by ascending sequence number.

The InstallUISequence table is processed in the order identified by the Sequence column (starting at sequence number 1), executing each action listed(Standard, Custom, and Dialog), as long as the Condition column for that action evaluates to true or is null (empty). The negative values in this table are "jumped" to on termination of the setup based on the result code of the installation. Once the "ExecuteAction" action is encountered, the InstallExecuteSequence is run. When the InstallExecuteSequence is complete, control will return to process the remaining UISequence table or it will jump to the corresponding negatively numbered sequence based on the result code of the installation process. Every action in the *UISequence tables are run in the currently logged on user's process for security reasons, and this is known throughout MSDN's documentation as the "client" portion of the installation.

The InstallExecuteSequence is where things get interesting, and differs based on platform.

• On Windows NT based systems, this table is evaluated and run in a separate process from that of the User Interface. See my earlier article on Properties for information as to what is passed from the "client" (UI portion) to the "server" or "service" portion of the installation. To recap, the admin user has two processes owned by him/her, one is the UI sequence processing, and the second is the Execute sequence processing. The non admin user installing with elevated privlidges can have three processes - the same two as above, plus a third process, owned by the System, which executes the deferred
  msidbCustomActionTypeNoImpersonate actions.
• On Windows 9x based systems, both the *UISequence and *ExecuteSequence and all actions contained therein are run in the same process.

Actions in the *ExecuteSequence should only use UI interactions that use the Session.Message() or MsiProcessMessage() API's and not reference the Dialog table or contain UI's (such as MessageBoxes, status bars, shell progress indicators, dialogs from 3rd party libraries, etc.) in order to respect the UILevel wishes of the user.

Now or Later

Let's continue the discussion by distinguishing the Immediate vs. Deferred custom action categories a bit more. An immediate action is one that is "run" or "executed" immediately when the action is encountered in the sequence. A Deferred action is one that, when encountered in one of the Sequence Tables, is written to a sequential "installation script" with some metadata - you can think of this as a "To-Do list". Wherever you see the words "script" or "written to the script" in MSDN documentation, this is what I am referring to. More on deferred actions and scripts later - let's get the Immediate Custom Actions out of the way first.

Immediate actions have a method of assuring they will only be executed a set number of times - this is perhaps the most confusing page of documentation in the MSDN library. The Custom Action Type msidbCustomActionTypeFirstSequence will cause the action to not run in the Execute sequence if it already ran in the User Interface sequence. The msidbCustomActionTypeOncePerProcess flag is similar to the msidbCustomActionTypeFirstSequence flag, with the exception that it will run an action again in the Execute sequence if the Execute Sequence is being run in a different process than the User Interface process (In English - it will run in both sequences on NT machines, and only once on 9x based machines). The msidbCustomActionTypeClientRepeat flag means if you run the MSI in silent mode (and therefore don't have the UI sequence table processed) , the Custom Action wouldn't execute, but if you ran it in full UI mode it would execute. The lack of any of these flags causes the action to be run always. Remember that Immediate Actions should not modify the state of the system in any way. Deferred actions cannot have the properties described in this paragraph. Deferred Actions are "recorded" in what is called the "installation script" before anything is actually run/executed. The script is created in the *ExecuteSequence when the InstallInitialize action is encountered, therefore deferred actions can only be sequenced after this action. The only time an item is written to the script is if the Condition is true at the time it is to be written to the script (as indicated in the Condition column of the Sequence table). In reality, there are two scripts being generated, the "installation script" and the "rollback script". I'm going to treat it as if there is only one script for simplified understanding.

Deferred Actions and the script

Think of this example: The MSI Engine is going through each of the actions in the InstallExecuteSequence in order of the sequence numbers, when - blammo - it hits the InstallInitialize action. At this point, it creates a data structure known as the "script" and starts recording all actions that meet the condition specified in the Condition column and are marked as msidbCustomActionTypeInScript along with some magical metadata (In other words it only records deferred actions in the script). If an immediate action with a proper condition (that also is compliant with the multiple execution flag setting) is encountered, it runs immediately. Otherwise, it keeps adding stuff to the script until it hits an InstallExecute, InstallExecuteAgain, or InstallFinalize action is encountered. Ignoring InstallExecute and InstallExecuteAgain for now, when InstallFinalize is encountered, it completes writing to the script and seals it from further additions. Then it begins processing the script by executing the actions, top down. The details of how the list is processed and the effect of InstallExecute and InstallExecuteAgain is covered in a bit.

So, what exactly is this magical metadata mentioned above? We already know that the condition column was evaluated before the decision was made to write the action to the "script" - if the condition was false, the task was not written. As near as I can figure, this metadata holds three major things:

• The value of a property that having the same name as the CustomAction - the so-called CustomActionData property you have undoubtably heard so much about. Additionally, the ProductCode and UserSID properties are available in a Deferred action.
• How to Run it flags (derived from the Type column of the CustomAction table)
• The type of action (DLL, EXE, Script, etc.) and the path to the CustomAction code.
  
• Impersonation type - The flag msidbCustomActionTypeNoImpersonate will cause the action to run in the system context vs. the otherwise usual currently logged-on user context. The exception to this rule is a server with Terminal Services which normally runs actions in the system context unless the msidbCustomActionTypeTSAware flag is set (flag valid on Windows 2003 Terminal Server only).
  
• msidbCustomActionTypeAsync indicates that once the action is kicked off there is no need for the MSI engine to wait for it to complete before executing the next action. However, the MSI engine does want to know the return code, and as such will wait around at the end of the sequence until it finally returns if the action takes that long. All actions are synchronous unless this flag is set. Only EXE type custom actions can also combine the msidbCustomActionTypeContinue flag so that the installation can exit even while the Custom Action EXE is still running asynchronously.
  
• msidbCustomActionTypeContinue indicates that the exit code of the custom action is ignored - therefore the sequence will progress even if the return code indicates failure.


• When to run it flags indicates three possible things (again derived from the Type column of the CustomAction table). As briefly mentioned earlier there are actually two different scripts written - Commit and Rollback actions are both written to a separate "Rollback Script", but ignore this for now.
• "Normal Deferred" assuming nothing has failed and the list is being processed top to bottom - msidbCustomActionTypeTSAware or msidbCustomActionTypeInScript
  options without msidbCustomActionTypeRollback or msidbCustomActionTypeCommit flags
• "Commit" - Run after the entire (sealed) script is processed once successfully (After InstallFinalize is encountered and the complete script has been processed once successfully). Actions in this category have themsidbCustomActionTypeTSAware or msidbCustomActionTypeInScript combined with msidbCustomActionTypeCommit flags in the Type column. This is only written if Rollback is not disabled - more on this later..
  
• "Rollback" - if another deferred action failed, this will be run. This is only written for the msidbCustomActionTypeTSAware or msidbCustomActionTypeInScript
  when combined with msidbCustomActionTypeRollback actions, and only if Rollback is not disabled - more on this later..

How the Script is Processed

To recap, we have a "script" containing deferred actions along with some metadata that we started collecting when InstallInitialize was encountered in the sequence. We can potentially start processing the list before it is completely written (assuming a InstallExecute or InstallExecuteAgain was encountered prior to InstallFinalize in the sequence tables).

Processing always begins top-down, and there are up to three "phases" in the script processing based on the "when to run it" metadata described above.

First, only "normal deferred" actions are processed. Each deferred action is executed only once - if an InstallExecute or InstallExecuteAgain action was encountered in the sequence tables causing a portion of the script to be executed prior to the script being completely written, "normal deferred" actions already executed are skipped.

If an error is thrown during the execution of a "normal deferred" action, the processing of the list reverses and starts moving backwards from where it currently is processing, executing the "On Rollback" actions in the list. Rollback can be "turned off" at a system level, if this is the case, Rollback actions will not be written to the script, and will not be executed.

Assuming all "normal deferred" actions are successful, processing of the script begins executing at the top again, this time executing the "On Commit" actions. Retuening a failure from a Commit Custom Action will cause any previously written Rollback actions to be executed. Because this causes some interesting and practically impossible to test scenarios, it is important to write solid Commit actions that do not fail or also specify the msidbCustomActionTypeContinue flag in the CustomAction table. Rollback can be "turned off" at a system level, if this is the case, Commit actions will also not be written to the script, and will not be executed.

Dealing with Rollback and Commit

If you are familiar with database concepts, Windows Installer is transactional. In other words, a failure to install a MSI package should leave the system in the exact state it was prior to beginning the installation. As such, Rollback and Commit actions are a part of the lexicon of Windows Installer, and understanding this when writing a Custom Action is essential. Also essential is understanding that Rollback can be disabled on a system level, thus, Rollback and Commit actions are not executed.

To adhere to the Windows Installer "Best Practices," all changes that are made to a system are in a "deferred" action - as an example, the "InstallFiles" standard action is actually a deferred action under the hood. Although not apparent in this manner (nor is apparrant by looking at the Sequence tables), the InstallFiles "normal deferred" action creates a backup of all the files it replaces if Rollback is enabled. This way, if an error occurs, all the replaced files can be restored to their previous state if an error occurs in a later deferred action through the Rollback process. The Commit actions are only run after all the "normal deferred" actions are run successfully - and the MSI engine uses the Commit action to delete the Rollback files.

Because of their nature, Rollback and Commit actions cannot be asynchronous.

A "normal deferred" Custom Action should only save rollback information if the RollbackDisabled property is not set. If it is set, Rollback and Commit actions will not be run, and this will cause your saved rollback data to remain after the installation is completed. You will need to pass this data into your "normal deferred" CustomActionData property in order to check if you should save rollback information.

Other Considerations

Inside a CustomAction, you may need to evaluate if a patch or repair is occurring. Since deferred CustomActions do not have the ability to check properties (other than the few properties mentioned earlier), you need to use an Immediate action to check (and pass on to the deferred action via CustomActionData) the value(s) of REINSTALL, PATCH, and/or MsiPatchRemovalList properties. For more details, see the Windows Installer Team Blog's comments on the subject or Heath Stewart's post.

To determine if you should save Rollback information in your deferred action, you will need to do the same technique as above to pass the value of the RollbackDisabled property. Other properties of interest for deferred CustomActions (depending on what they do) is ALLUSERS, and possibly UILevel.

To handle user requests to cancel the installation in any custom action, make sure that you check the return value from MsiProcessMessage() (or Session.Message()) calls to handle the IDCANCEL return value. I will be showing examples of handling progress messages in future posts on this topic. In the meantime check out the Windows Installer Team blog entry on the subject.

Another tricky concept is scheduling a reboot, if required, from a deferred Custom Action. Normally, you can't! To work around this oversight you have several options. The one I use is creating a file on the filesystem that the user running the installation can have delete access to. This is where the ProductCode and UserSID properties available to a CustomAction come in handy. Find the temp directory of the user using their SID and write a file with the name "ProductCode_rebootme" there. In an immediate action scheduled after InstallFinalize, check for the existance of this file and if present, delete it and call MsiSetMode() to indicate a reboot is required.

MSDN provides a few pointers on security here and here. In my opinion, it is extremely difficult to author a "data secure" MSI. If you need to pass internally sensitive data to a deferred custom action, use encryption on the CustomActionData in addition to adding sensitive properities (and the name of the deferred custom action) to the MsiHiddenProperties and using the msidbCustomActionTypeHideTarget CustomAction type flag. It is possible to digitally sign an installer package and its cabinet files. It is trivial to circumvent these measures if someone wanted to - and in a typical administrative installation, the msi signing is removed anyway. Think about data security in an MSI as glass in a jewelry store - it only keeps the honest people honest.

If a Custom Action needs to use disk space (to install a database, for example), you need to author (or modify at run time) the ReserveCost table. This is keyed on the installation state of a component.

The use of COM in a CustomAction is fine - when a dll-based Custom Action runs, it is created on its own thread. There is some conflicting documentation on this, however. MSDN claims CoInitialize is called in a per-machine installation and is not called in a per-user installation. A post here (scroll through to the comments) indicates that CoInitialize and/or its brethren CoInitializeEx is not called on this thread. I like MSDN's advice to not quit if it is determined the thread is already COM initialized.

Terminal Server is special. Throughout this article I have mentioned the *TSAware flags. In a per-machine installation on a Terminal Server, actions not marked with this flag are run as LocalSystem, where on a non-terminal server system they will normally impersonate the calling user.

The way patching and upgrading is handled, and its effect on Custom Actions, is the subject of a future article in this series.

The Ten Cent Summary

Custom Actions can be very powerful, and can be run in a variety of different ways and at a variety of times. The important things to remember when deciding on how to design them are:

• Immediate actions are normally used to "set up" deferred actions (by stuffing CustomActionData) or to simply evaluate the state of a machine. Immediate actions are NEVER to be used to alter the state of a machine.
• Deferred actions come in three different flavors, and where you have one, you should have all of them - the "normal deferred", rollback, and commit. The "normal deferred" should save undo information on the system somewhere if rollback is enabled. The rollback action will use this undo information to restore the initial state of the machine. The commit actions will clean up the rollback's undo information. Deferred actions can only be included in the execute sequences between InstallInitialize and InstallFinalize.
• Rollback only occurs if an error is encountered while processing deferred actions (including Commit actions) between InstallInitialize and InstallFinalize in the *Execute sequences.
• Because the conditions of deferred actions (including commit and rollback actions) are evaluated at the time the script is written, the same condition should be used for the sister actions.
• Actions should be written to run correctly regardless of the UI mode and respect the wishes of the user when it comes to UI levels.
• Never use a nested installation custom action type.
• Understand the flags and options for scheduling and running actions as presented above.

Part II of this series will cover writing our first Custom Action in C++!

InstallScript and Enumerators

I was helping a coworker out today trying to figure out why a simple VB Script using WMI could not be "ported" to InstallShield's InstallScript. The reason turned out to be that InstallScript does not support handling enumerators - In VBScript, you can enumerate a collection using the "For Each" construct. In JScript, you have the Enumerator object. In InstallScript - sadly - you have nothing.

I started thinking that I may be able to create a DLL to handle this. I then thought that someone else likely has done the same. My Google-foo came in handy and I found a post by __EDITED___ that covers the topic quite well. I was able to easily compile his C++ code into a DLL and use it immediately.

UPDATE: Turns out that the link previously referenced above was pretty much a blatent copy of code previously submitted to Installsite.org. Thank you to the commenter for pointing this out.

In the deleted link above, the poster was complaining about certain InstallShield limitations. I agree with most of the comments in his post - InstallShield's help system is at least second-rate. The InstallShield books by Baker are nice to have around, but not everyone has this luxury.

I disagree with his comments that MSI is "limited." It certainly is not the "holy grail" of installation technology - and I doubt that this can even exist. MSI technology is extremely extensible, but this comes at a cost of understanding the design patterns inherent in it - which are vastly different than the "other" installation technologies. I know I mentioned this before, but using MSI technology could practically eliminate vendor dependence and ease the switch between authoring tools.

Unfortunately, the documentation for MSI technology is equally as poor as InstallShield's. Rumor has it (plus the Amazon ratings) that Phil Wilson's book on the subject is pretty good - if Brian will ever let me borrow his copy I can give it an official review. I will attempt to help out with this by (finally) publishing my series on Custom Actions over the next several weeks.