Custom Action Tutorial Part II – Creating the Project

[Update 3/10/2009: Updated download link. Thanks for hosting, Aaron!]

In Part I we covered the dry material - what custom actions are, and how to use them. In this part of the tutorial, we will configure the development environment and create a baseline project to start from when developing future Custom Actions. Please make sure your system is configured as described in the first part before continuing.

I am providing a download link for the final project described in this part for convenience (see the end of this post for the link), but I recommend first time custom action developers follow the steps so they understand what is going on. For my more experienced readers, please bear with the simplicity here, but review the project settings before downloading or using the template in future parts.

Create a Basic Installation Project

To create our solution, open Visual Studio 2003. We are going to create two projects to start with – one a Win32 Project called "HelloWorld", and a deployment project that will create an MSI to install it. Go to the File->New->Project menu. Select the Visual C++ Projects->Win32->Win32 Project. Call the name of the project "Hello World" and place it into a directory such as C:\Projects\CATutorial. Select Finish in the Wizard dialog. When the IDE initializes, you should see the Solution Explorer window – Right-click on the Solution 'HelloWorld' line at the top of this box and select Add->New Project. Find the Setup and Deployment Projects category and select Setup Project, and select the Setup Project Wizard – you can accept the default name of Setup1. Hit next, and accept the defaults for page 2. On page 3, check the Primary Output from HelloWorld option and hit Finish. When the project opens, select the Users Programs Menu and right-click in the pane to the right. Select Create New Shortcut. Double-click Application Folder and double-click the Primary Output from HelloWorld (Active). Rename the shortcut if you wish.

Now we have both an EXE and an MSI project that installs the EXE plus a shortcut to it. At this point, we can build and test our MSI by right-clicking the Setup1 project and selecting Build. Next, right-click the project and select Install. You will be walked through the standard MSI project dialog sequence – just accept the defaults and finish the install. A shortcut is installed to Program Files in the Start Menu, which will open a window that pretty much does nothing. If you get this far, you are on the right track.

Add a DLL Project

To make a Custom Action DLL, we want to now add a new project – a Win32 Project that is a DLL. Follow the add project instructions for "HelloWorld" above, but call it "CustomAction" and in the Wizard, select the Application Settings and pick DLL for the Application Type. Press Finish. We now need to add some items to this project – please follow this closely.

1. Right-click on the CustomAction project and select Add->New Item. Find the Module-Definition File (.def) and select it. Call the file CustomAction and press Open. We'll add some content to it in a later step.
2. Right-click on the CustomAction project and select properties.
3. In the Configuration dropdown, select All Configurations
4. Expand the Linker folder, and select input. In the Additional Dependencies edit box, enter msi.lib and hit Apply. NOTE: As an alternate to this and the previous 2 steps, you can use the following directive in the stdafx.h or any other source file to tell the linker to link with msi.lib: #pragma comment(lib, "msi.lib")
5. In the upper right hand corner of the Property Pages dialog box, click Configuration Manager.
6. Under the Active Solution Configuration dropdown, select New. Enter the name Release Unicode, select Release from the Copy Settings from dropdown, and leave the checkbox checked. Press OK.
7. Repeat the previous step, calling the Configuration Debug Unicode and select Debug under the Copy Settings From dropdown. Close the Configuration Manager.
8. We are now back at the CustomAction property pages. Select Debug Unicode from the Configuration dropdown. Under the General Configuration Properties, find the Character Set setting and select Use Unicode Character Set. Press Apply.
9. Repeat the previous step for Release Unicode. Press OK to close the CustomAction property pages.
10. Repeat the previous two steps for the "HelloWorld" project by opening the property pages for the HelloWorld project.
11. In the CustomAction project, under the "Header Files" folder, double-click the file stdafx.h file to edit it. Under the // TODO: line, add the following lines:
    #include <tchar.h>
    #include <msi.h>
    #include <msiquery.h>
    #include <strsafe.h>
12. It is always a good idea to add versioning information to all the compiler outputs you build, so right-click on the CustomAction project and select Add->Resource. Pick Version from the list by double clicking on it. Edit the information here if you so desire.
13. In the CustomAction.cpp file, add four functions based on the following template named Install, Commit, Rollback, and Uninstall – function names are case sensitive.
    extern "C" UINT __stdcall Install(MSIHANDLE hInstall)
    {
    return ERROR_SUCCESS;
    }
14. Edit the CustomAction.def file and make it look like this:
    LIBRARY CustomAction
    EXPORTS
    Install
    Commit
    Rollback
    Uninstall

As a final step, we want to add the Custom Action DLL to the setup project. Right-click on Setup1 and select View->Custom Actions. Right-click on Custom Actions in the pane that opens up and select Add Custom Action. Double-click on Application Folder, and then click Add Output. Select CustomAction from the Project dropdown and accept the default Primary Output and Active Configuration settings. Click OK, then click OK on the Select Item in Project dialog. You will notice that four custom actions are added, and in viewing the properties for each of them the EntryPoint corresponds to the function names we created in the CustomAction.cpp file.

At this point, lets right-click on the Setup1 project and build it. You will notice that the file "msi.dll" is detected as a dependency for the Setup1 project – as a matter of fact, it will create a warning during compilation. Right-click on the msi.dll file under dependencies and check the Exclude option. Rebuilding the Setup1 project should now generate no warnings.

If you have not uninstalled the setup project from the earlier test, do so now, and run the Setup1 installation. You should see no difference from the previous time you ran the setup. If you get errors, you may want to download the zip file that contains the projects and steps we have performed up to here, and use WinDiff from the Bin directory of the Platform SDK to see if you can figure out the issue by comparing the directories of my project against yours.

Save your workspace and exit Visual Studio. Create a .zip file of the project directories – if you used my directory structure, you want to compress the "C:\Projects\CATutorial" directory. We will use this as a template for any future C++ custom action tutorials.

Initially, I was going to end this part here, but I thought people would be cursing me, saying that this whole exercise was pointless and no fun! After all this work, we created a Custom Action that did absolutely nothing! Let’s rectify that! Open your solution and replace the Install function in the CustomAction.cpp file with the following code (Don't worry if you don't understand it, we will cover this later):

extern "C" UINT __stdcall Install(MSIHANDLE hInstall)
{
PMSIHANDLE hRecord = MsiCreateRecord(0);
MsiRecordSetString(hRecord, 0, TEXT("Hello MSI!"));
MsiProcessMessage(hInstall, INSTALLMESSAGE(INSTALLMESSAGE_USER + MB_OK), hRecord);
return ERROR_SUCCESS;
}

When you rebuild the project and install it, you will see a "Hello MSI!" message when the Install Custom action runs.

You can download the Visual Studio 2003 Project Template here:

(Special thanks to Aaron Stebner for hosting this for me). The next installment in this series will explain what we did (and why we did it)!