Category: MSBuild

Automating creating NuGet packages with MSBuild

NuGet is a great way of shipping projects. You work on a project, you publish a package and it is immediately available to, literally, millions of developers. Creating a package consists of a few steps like authoring a .nuspec file, creating a folder structure, copying the right files to the right folders/subfolders and calling the nuget pack command. While the steps are not complicated they are error prone. I learnt this lesson when I shipped the first alpha versions of some of my NuGet packages. What happened was that I would create a package and then I would start feeling some doubts – did I really build the project before copying the files? did I copy the Release and not the Debug version? did I sign the file? And then people started using my packages and started asking (among other things) for a version that would work on other versions of .NET Framework. This meant that the amount of work to create the package would basically at least double since the steps I outlined above would have to be followed for each targeted platform. I had already decided to automate the process of creating NuGet packages but having to ship a multiplatform NuGet package was a forcing function to actually do the work. I set a few goals before starting working on this:

  • I will be able to create a package with just one simple command
  • I will be able to create a multiplatform package
  • I will be able to exclude/include platform specific code
  • The assemblies included in the package will be signed
  • I will be able to strip InternalsVisibleTo from my assembies
  • I won’t have to check any binaries in to the source control
  • None of the changes will break Visual Studio experience (i.e. I will be able to use Visual Studio the same way I was using it before the changes)

Since I am using Visual Studio for all the projects I ship NuGet packages for using MSBuild to achieve my goal was a no brainer. I started from enabling building the project for multiple .NET Framework versions (in my case I really needed only to be able to target .NET Framework 4 and .NET Framework 4.5). This was pretty straightforward – if you open your .csproj file you will quickly find the following line:

<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>

which, as you probably already guessed, indicates the target .NET Framework version. This can be parameterized as follows:

<TargetFrameworkVersion
Condition="'$(TargetFrameworkVersion)' != 'v4.0'">v4.5</TargetFrameworkVersion>

which will enable building the project for .NET Framework 4 just by passing/setting the TargetFrameworkVersion parameter to ‘v4.0’. If any other value is passed/set (or the value is not passed/set at all) the project will be built against .NET Framework 4.5. You can now test the project by building it from the developer command prompt. The following command will build the version for .NET Framework 4:

msbuild myproj.csproj /t:Build /p:TargetFrameworkVersion=v4.0

Note that the above command may fail. One of the reasons might be that your code uses APIs that are available only on .NET Framework 4.5 (async is probably the best example but there are many more). Therefore you may need a mechanism to exclude this code or replace it with a .NET Framework 4 counterpart. Typically this is done using the #ifdef precompiler directive. You just need a constant (let’s call it NET40) that will indicate that the code is being built against .NET Framework 4. We can define this constant in the csproj file depending on the value of the TargetFrameworkVersion property we already set.
To do that we just need to create a new PropertyGroup just below PropertyGroups used to set configuration (i.e. Debug/Release) specific properties. Here is how this new property group would look like:

 <PropertyGroup>
<DefineConstants
Condition=" '$(TargetFrameworkVersion)' == 'v4.0'">$(DefineConstants);NET40</DefineConstants>
</PropertyGroup>

Now we can use the NET40 in the #ifdef precompiler directives to conditionally compile/exclude code for a specific platform.
While we are at it we can take care of removing InternalsVisibleTo attributes from the code. We can use the same trick as we used for the TargetFrameworkVersion and define a constant if a property (let’s call it InternalsVisibleToEnabled) is set to false. By default its value will be set to true true by but when building a package (as opposed to building the project itself) we will set it to false. This will allow us to use the constant in the #ifdef directives to exclude code we don’t want when building packages. With this change the PropertyGroup created above will turn to:

<PropertyGroup>
<DefineConstants
Condition=" '$(TargetFrameworkVersion)' == 'v4.0'">$(DefineConstants);NET40</DefineConstants>
<DefineConstants
Condition=" '$(InternalsVisibleToEnabled)'">$(DefineConstants);INTERNALSVISIBLETOENABLED</DefineConstants>
</PropertyGroup>

Another important thing to look at are references. If you have a reference to an assembly that is not shipped with the .NET Framework you need to make sure that you are referencing a correct version of this assembly. This is especially conspicuous if you include other multiplatform NuGet packages – referencing a wrong version of the package may cause weird build breaks or, some APIs you would expect to be present will appear to be missing. This can be fixed with conditional references. For instance in some of my projects I am referencing the EntityFramework NuGet package and I need to reference the correct version of EntityFramework.dll depending on the .NET Framework version I compile my project against. To solve this I can use the MSBuild Choose construct like this:

<Choose>
<When Condition="'$(TargetFrameworkVersion)' == 'v4.0'">
<ItemGroup>
<Reference Include="EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\packages\EntityFramework.6.1.0\lib\net40\EntityFramework.dll</HintPath>
</Reference>
<Reference Include="EntityFramework.SqlServer, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\packages\EntityFramework.6.1.0\lib\net40\EntityFramework.SqlServer.dll</HintPath>
</Reference>
</ItemGroup>
</When>
<Otherwise>
<ItemGroup>
<Reference Include="EntityFramework, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\packages\EntityFramework.6.1.0\lib\net45\EntityFramework.dll</HintPath>
</Reference>
<Reference Include="EntityFramework.SqlServer, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL">
<SpecificVersion>False</SpecificVersion>
<HintPath>..\packages\EntityFramework.6.1.0\lib\net45\EntityFramework.SqlServer.dll</HintPath>
</Reference>
</ItemGroup>
</Otherwise>
</Choose>

That’s pretty much it as far as building the project against different versions of the .NET Framework goes. With this we can look at the NuGet side of things. The first thing to do is to open the solution in the Visual Studio, right click on the solution node in the project explorer and select “Enable NuGet Package Restore”. This will do a few things

  • it will enable restoring missing NuGet packages when building. This is very useful and help prevent from having binaries checked in in the source control (you almost always want it)
  • to achieve the above it will create a .nuget folder which will contain a few files with the NuGet.targets being the most important for us
  • it will modify .csproj files to define some properties but most importantly it will include the NuGet.targets file

Note: one of the files NuGet drops to the .nuget folder is Nuget.exe. If you are using source control (if you are not you deserve to be named a lead developer of a new feature in this old COBOL project no one has touched in years – they did not use source control either so you should be fine) you want to check in all the files from the .nuget folder but Nuget.exe.
Save your files or, even better, close Visual Studio. This is important. If you modify your project files both inside and outside VS you will lose changes you made outside VS in the best case (and only if you pick the right option when VS detects you edited files outside VS). In the worst case you will end up in this .csproj limbo when .csproj files are only partially modified by tools and you are not able to recreate what has been lost. The tools no longer work (or work randomly which is even worse) because some changes are lost, the project does not compile and the best option is just to revert all the changes to the last commit (if you are not using source control then, you know, the COBOL project needs a lead (or, actually, any) developer) and start from scratch.
Now we are ready to add the logic to build the NuGet package. We will create a new target called ‘CreatePackage’ (I originally wanted to call it ‘BuildPackage’ but it turned out that a target with that name already exists in the .NuGet.targets file) which will be the entry point to build the package. Currently all my projects I ship NuGet packages for are simple and contain just one assembly with the product code and one assembly with tests. Since tests are not part of the NuGet packages I ship I can add the CreatePackage target to the .csproj file containing the product code. If I had more assemblies I would create a separate file (probably called something like build.proj where I would have all the targets needed to build and package all the assemblies). In this target I would need to:

  • build my assembly/assemblies for each platform I want to target
  • create a folder structure required by NuGet
  • copy artifacts built in the first step to folders from the second step
  • create the package using NuGet.exe pack command

The first thing is to further modify the .csproj file to fix a couple of problems. The first problem we need to fix is that each time we build the project we overwrite existing files. Normally (e.g. when working from Visual Studio) it is not a problem but because we are going to invoke build more than once (we need to build for multiple platforms) subsequent builds would overwrite files built by previous builds. We could solve this by just copying files between builds but the solution I like better is just to be able to tell the build where to place the build artifacts. This can be done by removing setting the OutputPath property in the configuration specific PropertyGroups (e.g. to ‘bin\Release\’) and adding the following conditional OutputPath property definition to the first PropertyGroup after the Configration. The new OutputPath definition would like this:

<OutputPath Condition="'$(OutputPath)' == ''">bin\$(Configuration)\</OutputPath>

This allows to pass the OutputPath value as a parameter and if it is not empty it will be used throughout the build. Otherwise we will set a default value which effectively will be the same as the one that would have originally been set.
The second thing to take care of is the case where there is no NuGet.exe file in the .nuget folder (I recommended to not check this file in so it will be missing for newly cloned repos or when you clean the repo with git clean –xdf). This can be easily fixed by adding the following property definition to the first PropertyGroup:

<DownloadNuGetExe>true</DownloadNuGetExe>

Now whenever you invoke a target that depends on the NuGet’s CheckPrerequisites target NuGet will check if the NuGet.exe file is present and if it is not it will download it.
With the above changes we are ready for the CreatePackage target. Here is how it looks like (I copied it from the Interactive Pre-Generated Views project):

<Target Name="CreatePackage" DependsOnTargets="CheckPrerequisites">
<Error Text="KeyFile parameter not spercified (/p:KeyFile=MyKey.snk)" Condition=" '$(KeyFile)' == ''" />
<PropertyGroup>
<Configuration>Release</Configuration>
<PackageSource>bin\$(Configuration)\Package\</PackageSource>
<NuSpecPath>..\tools\EFInteractiveViews.nuspec</NuSpecPath>
</PropertyGroup>
<RemoveDir Directories="$(PackageSource)" />
<MSBuild Projects="$(MSBuildThisFile)" Targets="Rebuild" Properties="InternalsVisibleToEnabled=false;SignAssembly=true;AssemblyOriginatorKeyFile=$(KeyFile);TargetFrameworkVersion=v4.5;OutputPath=bin\$(Configuration)\net45;Configuration=$(Configuration)" BuildInParallel="$(BuildInParallel)" />
<MSBuild Projects="$(MSBuildThisFile)" Targets="Rebuild" Properties="InternalsVisibleToEnabled=false;SignAssembly=true;AssemblyOriginatorKeyFile=$(KeyFile);TargetFrameworkVersion=v4.0;OutputPath=bin\$(Configuration)\net40;Configuration=$(Configuration)" BuildInParallel="$(BuildInParallel)" />
<Copy SourceFiles="bin\$(Configuration)\net45\$(AssemblyName).dll" DestinationFolder="$(PackageSource)\lib\net45" />
<Copy SourceFiles="bin\$(Configuration)\net40\$(AssemblyName).dll" DestinationFolder="$(PackageSource)\lib\net40" />
<Copy SourceFiles="$(NuSpecPath)" DestinationFolder="$(PackageSource)" />
<Exec Command='$(NuGetCommand) pack $(NuSpecPath) -BasePath $(PackageSource) -OutputDirectory bin\$(Configuration)' LogStandardErrorAsError="true" />
</Target>

Let’s look at this target line by line since there are some small additions I have not mentioned yet. One of my goals was to be able to sign the assembly. I do it by passing a path to my .snk file. To make sure I don’t build packages with non-signed assemblies I error out on line #2 if the path to the file was not provided (the error also tells me the parameter name so that I don’t have to open the .csproj file each time I need to build the package to remember the name of the parameter used to pass the path to the .snk file). Then I define a few properties I use later:

  • Configuration – I always want to ship Release versions in NuGet packages so it is hardcoded to ‘Release’
  • PackageSource – the path where the target will create folders that will be later consumed by NuGet. Platform specific assemblies will be placed in subfolders
  • NuSpecPath – the path where the .nuspec file lives

After that I build the project. I do it twice – once for each target platform. You can see that when you look at the parameters passed to the Build target – especially the TragetFrameworkVersion (‘v4.5’ vs. ‘v4.0’) and the OutputPath (‘net45’ vs. ‘net40’). SignAssembly and AssemblyOriginatorKeyFile make sure that the assembly will be signed. InternalsVisibleToEnabled is set to false to exclude InternalsVisbileTo attributes. Once the assemblies are build they are copied to the folders the NuGet package will be built from. Note that the Copy MSBuild task will create the target folder if it does not exist. The only thing remaining is to create the NuGet package (at long last!). I execute the NuGet.exe (disguised as $(NuGetCommand) – a variable defined in the NuGet.targets file) with the pack option and provide the path to the .nuspec file, the folder structure to build the package from (the BasePath parameter) and the path to save the NuGet package to. The LogStandardErrorAsError attribute is telling MSBuild to fail the build if the executed command returns a non-zero exit code.
Now I can build my NuGet packages with the following command (from the developer command prompt):
msbuild myproject.csproj /t:CreatePackage /p:KeyFile=MyKey.snk
Despite all the manual edits to the .csproj file Visual Studio continues to work (at least at the same level it did before) which was the last of my goals.
This is it! With this guide you should be able to automate building your NuGet packages. Just in case here are links to some of the changsets I introduced this method in:

Once you understand what’s going on and have the template I provided above it takes less than 10 minutes to adapt it to your project (funnily, even coming up with the first version took me considerably less time (and wine) than describing it in this blog post).
Enjoy (and show me your package)!

Safe Git Clean with MSBuild

I like to clean my source tree once in a while. Unfortunately msbuild {whateverproject}.csproj /t:Clean often falls short – for more complicated solutions with multiple projects some temporary files generated during the build almost always are left on the disk. Removing these should not be a big deal given that the important files are under source control – I can simply remove everything that is not tracked. With git you do that with git clean -xfd command and the files are gone. Usually, this is the (sad) moment when I realize that there was that file you forgot to add to the index or this directory that was not supposed to be added to the index but had some “temporary” data I have been using for a couple of weeks now. To save myself a grief I came up with an MSBuild target that saves everything that is going to be deleted to a backup directory before the clean up. This way even if I missed something not everything is lost – I can go to the backup directory and restore the file.
The target itself turned out quite simple. The biggest problem was to make git commands work with MSBuild. It’s possible to run git commands with Exec task. However Exec task does not capture results of the command. The only way I found to capture the results was to redirect the command output to a file and then read the file contents to an ItemGroup with ReadLinesFromFile task. Not ideal for sure but seems to work.
Using the target is easy as the target takes only two optional parameters – BackupDir and DeleteBackupDir. BackupDir is the path to a directory where files that are not tracked should be saved. If the value is not provided files will be saved to a subfolder of the %TEMP% folder. DeleteBackupDir tells whether to delete the BackupDir if one exists – ‘false’ is the default value. An exemplary command would be:
MSBuild {projectfile} /t:SafeGitClean /p:BackupDir=C:\temp\backup /p:DeleteBackupDir=false
The target itself looks as follows:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="SafeGitClean">
    <PropertyGroup>
      <BackupDir Condition="'$(BackupDir)' == ''" >$(Temp)\SafeGitClean$([System.Guid]::NewGuid())</BackupDir>
      <DeleteBackupDir Condition="'$(DeleteBackupDir)' == ''">false</DeleteBackupDir>
      <TempFile>$(Temp)\SafeGitClean_UntrackedFiles$([System.Guid]::NewGuid())</TempFile>
    </PropertyGroup>
    
    <Message Text="$(BackupDir)" />

    <RemoveDir 
      Condition="'$(DeleteBackupDir)' == 'true' And Exists('$(BackupDir)')" 
      Directories="$(BackupDir)" />
    
    <Exec Command="git ls-files --other &gt; $(TempFile)" /> 

    <ReadLinesFromFile File="$(TempFile)">
      <Output TaskParameter="Lines" ItemName="UntrackedFile" />
    </ReadLinesFromFile>
        
    <ItemGroup>
      <UntrackedFileFixed Include="@(UntrackedFile->Replace('/', '\'))" />
    </ItemGroup>
    
    <Delete Files="$(TempFile)" TreatErrorsAsWarnings="true" />
      
    <Copy 
      SourceFiles="@(UntrackedFileFixed)" 
      DestinationFiles="@(UntrackedFileFixed->'$(BackupDir)\%(Identity)')" 
      OverwriteReadOnlyFiles="true" />
      
    <Exec Command="git clean -xfd" />      
  </Target>

</Project>

You can also find it in my github repo: https://github.com/moozzyk/MSBuild-Tasks

Pawel Kadluczka

MSBuild Zip task without external dependencies

I have seen quite a few build systems in my life. Most of them were very complicated with a ton of crap a lot of tools, dependencies, perl scripts, batch files, nested targets files and God knows what else. Figuring out how something worked (or, more often, why something did not work) was time consuming and very frustrating. One of the reasons for this was people were adding some stuff to the build system but no one has ever removed anything. What was even more annoying was that new dependencies oftentimes added tens of new files just to enable one small thing. Not only the enlistments were huge (how about ~200GB without QA tests?) but also configuring the machine to be able to build all this and run the tests was a sort of black magic. Because of all this I always felt bad about the fact that MSBuild did not have a Zip task available out-of-the-box. I feel that not having this one thing is the first step to having a build system that everyone hates. Yes, I know there wasn’t a Zip library in .NET Framework until now. Yes, I know there are third party Zip libraries out there. Yes, I know about MSBuild Community Tasks. Yes, I know MSBuild can somehow zip files internally as it can create VSIX files which are zip files… oops – this probably was not the best example. Anyways, not having a built-in Zip tasks means that you need to add some dependencies to your build system to be able to build your project. This will lead to a build system that no one wants to touch to not break anything. What about sharing just a single project? Like for instance my Code First view gen templates? Is it OK to tell people – “you can build it on your own – here is the project, but first you need to install this and this and this or it won’t build”? I don’t think it’s OK, but before I could not help much. Fortunately, a Zip library was finally added to .NET Framework 4.5. This allowed me creating my own Zip task. How is this task different from, for instance, the Zip task from the MSBuild community tasks? I created it as an inline task. As a resull it is just a small text file I can import to my projects. It can be checked in to my source control. It does not require any additional components being installed or present on the machine apart from what’s already there. If I need to know what the task is doing I can see the source without Reflector. I can easily change the task without having to recompile half of my build system just to be able to build what I actually want to build. (See the Disclaimer at the bottom of the page). The task looks just like this:

<UsingTask TaskName="Zip" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
    <ParameterGroup>
      <InputFileNames ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="true" />
      <OutputFileName ParameterType="System.String" Required="true" />
      <OverwriteExistingFile ParameterType="System.Boolean" Required="false" />
	</ParameterGroup>
    <Task>
      <Reference Include="System.IO.Compression" />
      <Using Namespace="System.IO.Compression" />
      <Code Type="Fragment" Language="cs">
      <![CDATA[        
        const int BufferSize = 64 * 1024;

        var buffer = new byte[BufferSize];
        var fileMode = OverwriteExistingFile ? FileMode.Create : FileMode.CreateNew;

        using (var outputFileStream = new FileStream(OutputFileName, fileMode))
        {
          using (var archive = new ZipArchive(outputFileStream, ZipArchiveMode.Create))
          {
            foreach (var inputFileName in InputFileNames.Select(f => f.ItemSpec))
            {
              var archiveEntry = archive.CreateEntry(Path.GetFileName(inputFileName));

              using (var fs = new FileStream(inputFileName, FileMode.Open))
              {
                using (var zipStream = archiveEntry.Open())
                {
                  int bytesRead = -1;
                  while ((bytesRead = fs.Read(buffer, 0, BufferSize)) > 0)
                  {
                    zipStream.Write(buffer, 0, bytesRead);
                  }
                }
              }
            }
          }
        }        
      ]]>
      </Code>
    </Task>
  </UsingTask>

Using the task is simple. Put the task to a separate file and import the file to the csproj file. In fact the file is available in my github repo – https://github.com/moozzyk/MSBuild-Tasks. Once you import the file to the project you just invoke the task as you would invoke any other task – for example (this is an actual except from one of my csproj files):


  <Import Project="common.tasks" />
  
  <Target Name="BeforeBuild">
    <ItemGroup>
      <FilesToZip Include="$(ProjectDir)\PayloadUnzipped\*.*" />
    </ItemGroup>
    <Zip 
      InputFileNames="@(FilesToZip)"
      OutputFileName="$(ProjectDir)$(TargetZipFile)"
      OverwriteExistingFile="true" />
  </Target>

That’s pretty much it. Works for me and hopefull will work for you.

Disclaimer:
I am not trying to diminish MSBuild Community Tasks or claim that inline tasks will solve all problems of this world. I am trying to say that for small simple tasks inline tasks can be just much more convenient.