Creating and using strong named assemblies in ASP.NET 5

The support for signing assemblies was first introduced to ASP.NET in the beta-6 release. In the upcoming RC1 release the way signing works has changed a little bit. In addition, from now on, assemblies that are part of ASP.NET 5 will be signed as well. Some people are questioning the decision of signing ASP.NET assemblies with strong name but there are good reasons to do it. First, some companies have a policy that all the assemblies they build must be signed. Without signing ASP.NET 5 could not be used in such environments since a signed assembly cannot depend on a non-signed assembly. Another reason is servicing – while it was never a goal to install ASP.NET 5 in GAC having signed assemblies leaves the option to use GAC to service ASP.NET 5 (e.g. in case of a critical security vulnerability) open.
There is a discussion about signing ASP.NET 5 assemblies where you can comment on the decision or ask questions.

In ASP.NET 5 there are three ways an assembly can be signed:

  • an assembly can be signed with a private and public key pair – this is just regular signing everyone knows and “loves”; assemblies signed like this can be installed in GAC etc.
  • an assembly can be delay signed – in this case the assembly is signed only with a public key but is really unusable unless you enable skipping strong name verification for this assembly using the sn tool (sn -Vr). The goal is to enable using assemblies signed with a private key to which developers don’t have access in test/dev environment.
  • an assembly can be OSS signed – this is a new way of signing that is very similar to delay signing in that the assembly is signed only with a public key. The difference is that it is no longer required to skip strong name verification for this assembly. The assembly just appears to be strong named while it really is not. (Internally OSS signing is oftentimes referred to as “fake signing” – in the Roslyn github repo there is even a tool called FakeSign). There are some scenarios that OSS signed assemblies cannot be used in – for instance an OSS signed assemblies cannot be installed in GAC. You can find more details on OSS signing and its limitations here.

If you want to sign an assembly in ASP.NET 5 you need to set appropriate compilation options in the project.json file. There are three settings that control signing:

  • keyFile
  • delaySign
  • useOssSigning

Depending on what combination of the signing options you use and the platform you are compiling on your assemblies will be signed in one of the ways enumerated above.
If you have just the keyFile in your compilation options (e.g. "compilationOptions": { "keyFile": "myKey.snk" }) the assembly will be signed with a private and public key pair (i.e. good (?), old signing) if you compile your project using Desktop Clr. CoreClr and Mono don’t support signing with a key pair so in these cases the public key will be extracted from the .snk file and will be used to OSS sign the assembly.
If you have both the keyFile and useOssSigning (e.g. "compilationOptions": { "keyFile": "myKey.snk", useOssSigning: true }) the public key will be extracted from the .snk file and the assembly will be OSS signed with the public key regardless of the runtime you use for compilation.
If you have just useOssSigning set to true, the assembly will be OSS signed with a hardcoded public key that is part of the ASP.NET 5 runtime.
If you set both useOssSigning and delaySign to true, the useOssSigning wins and the assembly will be OSS signed.
The delaySign option is ignored if you are compiling with CoreClr or Mono or you the keyFile option is missing.

Side note: When we enabled signing in beta-6 I noticed that there was a common misconception that signing depends on the platform the code was compiled for and not the platform the code was compiled on. You could tell this because people used the keyFile and (now removed) the strongName options together to sign assemblies produced for desktop Clr using the key pair (.snk) and OSS sign assemblies for CoreClr (since signing with a key pair is not supported on CoreClr). This resulted in errors since these options were mutually exclusive. The thing is that assemblies are signed when they are compiled and to compile an assembly you just use one runtime. Once an assembly is compiled and signed it will work with any (desktop Clr, CoreClr or Mono) runtime. In other words – if you sign an assembly with the public/private key pair (which you can do only if you compile the assembly using Desktop Clr) it will run just fine on CoreClr or Mono (they basically don’t verify strong names). From this point of view using the keyFile and strongName options together did not make much sense since it was just like saying “I want this assembly to be signed with a key pair and also with a public key only). This confusion was also one of the reasons why the strongName option was replaced with the useOssSigning option – the new option can be used together with the keyFile option and is more powerful.

Being able to OSS sign assemblies with an .snk opens up a couple of very interesting scenarios. For instance, imagine an open source project where the assemblies are signed using a private and public key pair during build. The .snk file is part of the project and is checked in to the source control. This basically means that the project was meant to be compiled using Desktop Clr. Now, if you wanted to compile this project with Mono or CoreClr things will most likely break – signing with a key pair did not work on Mono or CoreClr (before ASP.NET 5 RC1) so the assemblies will not be signed at all. This in turn may lead to further failures – e.g. if you use the InternalsVisibleToAttributes they will have to be modified as well to remove the public key. So, before you even started working on a project you had to spend a lot of time to introduce changes that will have to be thrown away when you commit your actual changes. OSS signing with .snk files fixes this because now without any changes to the project the assembly will be automatically signed with the public key and all code that depend on this assembly will just work.
The second scenario is similar to the first one except that this time the .snk file is not part of the project. So, you can clone and build the project (let’s call it A) but the assembly will not be signed. This is fine if assemblies that depend on A don’t have to be signed. But if they do you would have to sign A. The problem is that you don’t have the .snk so you will have to create your own .snk. As a result all assemblies that use A now need to be changed and recompiled because the identity of A changed. This might be quite a task but is doable… if you have code for all these assemblies. If you don’t then you’re dead in the water – you won’t be able to do that. Again, OSS signing can fix this scenario. Just dump the public key from the original assembly to an .snk (with sn -e [assembly] [targetfile]), build A and OSS sign it with the public key you just dumped. Everything else will just work (less than known limitations of OSS signing like installing OSS signed assemblies to GAC etc.) because the assembly will have the same identity as the original one.

Some miscellaneous notes:

  • assemblies signed with a private and public key pair can depend on OSS signed assemblies
  • to distinguish an OSS signed assembly from a delay signed assembly run sn –Vf [assembly]. sn -Vf will return the following error for OSS signed assemblies: Failed to verify assembly -- Strong name verification failed.
  • if you use xUnit as your testing framework you won’t be able to run tests on OSS signed assemblies with default settings. You will run into a limitation of OSS signing where the assembly cannot be loaded into AppDomain where shadow copying is turned on. To fix this run the xUnit runner with the –noshadow switch

Using native libraries in ASP.NET 5

In ASP.NET 5 RC1 we are adding a built-in support for native libraries. This may seem a little surprising – all in all ASP.NET 5 is all about managed code running on different platforms. There are, however, scenarios where this managed code needs to talk to unmanaged code. You can find such examples in ASP.NET 5 itself – the Kestrel Http server is build on top of the libuv library and Entity Framework ships a SQLite provider that uses the SQLite library. Before adding support for native libraries developers had to load native libraries on their own and then manually find and expose exported functions. Given the differences between platforms (Windows vs. non-Windows), architectures (x86 vs. x64 vs. ARM) and, possibly, runtimes (desktop Clr vs. CoreClr vs. Mono) the code to achieve the task was non-trivial. With the new feature the idea is that native libraries are shipped in a NuGet package, are part of a referenced project or live in a location where they can be automatically loaded from (e.g. /usr/lib) and the user just uses the DllImport attribute to get access to functions exported by the native library like this:

[DllImport(“mylib”)]
public static extern int get_number();

This is the simplest way of using the DllImport attribute. The name of the native library is passed as the parameter to the DllImport attribute while the exported function will be matched by convention using the name of the method attributed with the DllImport attribute. One important thing to note is that the name of the native library does not contain any filename extension. Both CoreClr and Mono will try appending different filename extensions if they can’t find a library with the exact name provided by the user and they will eventually use the extension specific for the given OS (Mono may also prepend the name with “lib”). Mono on OS X requires using __Internal as the name of the library – see below for more details.

Now, that you know how to consume native libraries in ASP.NET 5 let’s take a look at how to create a NuGet package that contains native libraries that can be consumed using the DllImport attribute. The most important thing is the package structure. Native libraries need to located in the $/runtimes/{runtime-id}/native folder where $ is the root of the package and the {runtime-id} describes the target platform the library is supposed to run on. Runtime Ids is a vast topic and I won’t go into a lot of details here – especially that, currently, they are not fully baked in when using native libraries and project references. For our purposes we can assume that a runtime id consists of the operating system name, version and architecture. For working with native libraries effectively it is enough to know the following runtime ids:

  • win7-x64
  • win7-x86
  • win7-arm
  • osx.10.9-x64
  • osx.10.10-x64
  • osx.10.11-x64

You can find a few oddities in this list. First a runtime id for Linux is missing. This is because you can’t really ship an .so that would work on any distribution of Linux in a package. While there are runtime ids that target selected distros (e.g. ubuntu.14.04-x64) or even just vanilla Linux (linux-x64) the recommended way of using native libraries on Linux is to have the user install the .so in a location from where it can be loaded automatically (e.g. install into /usr/lib or set the LD_LIBRARY_PATH environment variable accordingly). The second oddity is win7-arm. This seems odd because there was never an ARM version of Windows 7. The super-correct runtime to use here is actually win10-arm (ASP.NET 5 is not supported on Windows RT which would be win8-arm or win81-arm). However, if you used win10-arm you would need to remember to explicitly specify the runtime id each time you restore the package or publish your project. If you did not, you wouldn’t get your native library since without providing the runtime id packages will be restored for win7-{arch} (win7-arm in this case). So, because there aren’t any ASP.NET 5 runtimes that can run on pre-Windows 10 version of Windows it is safe to use win7-arm instead of win10-arm and it is much less troublesome. The last oddity is multiple runtime ids for osx. Currently, if you want to support multiple versions of OS X (officially CoreClr supports only OS X 10.10) you need to specify each version separately. Even if you use the same .dylib across all the versions it needs to be copied for each version of OS X you want to support.
For project references you need to use the same folder structure as for packages except that your root is now your project folder (i.e. the folder where the project.json lives).
The dnx repo contains sample projects for testing packages and project with native libraries. This is a good reference point if you ever get stuck.

What’s been covered so far should be enough to get started with native libraries in ASP.NET 5. You now know how to bind to functions exported from native libraries and how to build NuGet packages or structure projects containing native libraries. If you would like to understand how things are implemented under the cover (or need to know how to troubleshoot things) read on.
Loading native libraries works differently on different operating systems. In addition, different runtimes (CoreClr, desktop Clr, Mono) handle native libraries differently as well. CoreClr is consistent across all the platforms – whenever a native library needs to be loaded CoreClr will call ASP.NET 5 first and ask to provide the library. If the library CoreClr is asking for is in one of the referenced packages or projects ASP.NET 5 runtime will load it using a function exposed by CoreClr that allows to load a native library from a path. If you turn on tracing you will see the following trace entries when a native library is loaded:

Information: [LoaderContainer]: Load unmanaged library name=nativelib
Information: [PackageAssemblyLoader]: Loaded unmanaged library=nativelib in 1ms

On non-CoreClr runtimes things are a bit messy because non-CoreClr runtimes keep loading of native libraries to themselves. As a result it takes some tricks to help them load libraries from the right place. On Windows the LoadLibrary WINAPI function searches various locations when looking for a library. The search locations include folders specified in the %PATH% environment variable. So, when running on desktop Clr ASP.NET 5 will prepend the %PATH% environment variable with paths to native libraries. Note that one of disadvantages of this approach it is possible to hit the path length limit if a project references many packages with native libraries and this will most likely cause issues down the road. In traces this looks like this:

Information: [PackageDependencyProvider]: Enabling loading native libraries from packages by extendig %PATH% with: ;C:\Users\moozzyk\.dnx\packages\NativeLib\1.0.0\runtimes\win7-x86\native

On non-Windows systems the trick with the path does not work. While there is an environment variable that can be set to point the runtime linker to load a library from a non-default location (LD_LIBRARY_PATH on Linux and DYLD_LIBRARY_PATH on OS X) it has to be set before the process starts. The runtime linker reads and caches the value of the *_LIBRARY_PATH environment variable when the process starts and any further modifications have no effect. To work around this ASP.NET 5 on Mono on OS X will pre-load native libraries. This works but has one drawback – the name passed to the DllImport attribute can no longer be the library name but has to be __Internal to tell mono to look among already loaded libraries. This is ugly because it requires maintaining two sets of DllImport attributes – one that is specific to Mono on OS X with __Internal as the name and one that has the dll name which works everywhere else. If you want to see this ugliness at work you can take a look at Kestrel or the project for testing the DllImport attribute. In this case the traces will look as follows:

Information: [PackageDependencyProvider]: Attempting to preload: /Users/moozzyk/.dnx/packages/NativeLib/1.0.0/runtimes/osx.10.11-x64/native/nativelib.dylib
Information: [PackageDependencyProvider]: Preloading: /Users/moozzyk/.dnx/packages/NativeLib/1.0.0/runtimes/osx.10.11-x64/native/nativelib.dylib succeeded

The pre-loading trick does not work on Mono on Linux. In this scenario the user must make sure that the library can be loaded by the runtime linker. I don’t think this is a big problem – as explained above distributing .so’s in NuGet packages is not that a great idea anyways.

If you are trying to use a native library and ASP.NET 5 cannot find/load the library there are a few things you can try. First turn on tracing and check if you see traces similar to shown above. If you don’t see them and you are using a package reference most likely the native library was not restored correctly. To check this open the project.lock.json file and find a corresponding target section that contains the both the runtime and the runtime id (i.e. you are looking for "DNXCore,Version=v5.0/osx.10.11-x64" not "DNXCore,Version=v5.0"). Inside find the package description for your package with native libraries – e.g.:

"NativeLib/1.0.0": {
  "type": "package",
  "dependencies": {
    "System.Runtime": "4.0.21-beta-23506"
  },
  "compile": {
    "lib/dnxcore50/NativeLib.dll": {}
  },
  "runtime": {
    "lib/dnxcore50/NativeLib.dll": {}
  },
  "native": {
    "runtimes/osx.10.11-x64/native/nativelib.dylib": {}
  }
},

If the "native" does not exist, is empty or does not contain a correct path to your library this is the problem. In most cases the issue is with the runtime id – either the packages were not restored using the correct/expected runtime id or the package containing the native library did not have library for the given runtime id.
If you are seeing problems with loading native libraries on Mono you can try using the MONO_LOG_LEVEL environment variable. Setting it to debug will print a lot of traces which can be helpful troubleshooting issues. You can also filter out some categories. More details here.
Another helpful environment variable useful for debugging problems with loading libraries on Linux is LD_DEBUG. Again, if you turn on all tracing you will get a lot of stuff. You can however filter out things. This post contains a nice summary of available options.
Looks like on Mac you should be able to use DYLD_PRINT_LIBRARIES (I did not have to so far).
On Windows you can use Process Monitor to see what files are being accessed.
Also, remember that loading a dynamic library will fail if it depends on a library that cannot be found. You can check dependencies with ldd on Linux, otool -L on Mac OS X and dumpbin on Windows.

So, this more or less how native libraries can be used in ASP.NET 5 and how you troubleshoot issues.

Entity Framework View Generation Templates Updated for Visual Studio 2015

Entity Framework 7 may be around the corner and when it ships view generation will no longer be a concern because EF7 is not using views to abstract the database anymore. As of now, however, EF7 is still in beta and – as per the schedule – it will take another few months for it to be released. Even when it ships I don’t think many existing applications will ever move to EF7. In fact, there is still a lot of applications using EF5 or even EF4 that were never moved to EF6 and moving a pre-EF7 based app to EF7 may be even more difficult. In the meantime, Visual Studio 2015 was released. And while people not necessarily want to move their apps to use newer versions of dependencies many of them want to move to the latest available tooling. To make it possible I updated all the view generation templates so that they work with VS 2015. So, if you use or plan to use view generation templates in your applications download the latest version(s) from Visual Studio Gallery or install them directly from within Visual Studio. (Note that view generation templates in existing projects will continue to work in Visual Studio 2015 without this update. This update is required if you needed to add a new template to a project).

C++ Async Development (not only for) for C# Developers Part V: Cancellation

In the previous part of the C++ Async Development (not only for) for C# Developers we looked at exception handling in the C++ Rest SDK/Casablanca. Today we will take look at cancellation. However, because in C++ Rest SDK cancellation and exceptions have a lot in common I recommend reading the post about exception handling (if you have not already read it) before looking at cancellation.

C++ Rest SDK allows cancelling running or scheduled tasks. Running tasks cannot, however, be forcefully cancelled from outside. Rather, an external code may request cancellation and the task may (but is not required to) cancel an ongoing operation. (This is actually very similar to how cancellation of tasks works in C# async.)

So, cancellation in its simplest form is about performing a check inside the task to determine if cancellation was requested and – if it was – giving up the current activity and returning from the task. While conceptually simple, implementing a reusable mechanism that allows to “perform a check to determine if cancellation was requested” is not an easy task due to threading and intricacies around capturing variables in lambda expressions. Fortunately, we don’t have to implement such a mechanism ourselves since C++ Rest SDK already offers one. It is based on two classes – pplx::cancellation_token and pplx::cancellation_token_source. The idea is that you create a pplx::cancellation_token_source instance and capture it in the lambda expression representing a task. Then inside the task you use this pplx::cancellation_token_source instance to obtain a pplx::cancellation_token whose is_cancelled method tells you if cancellation was requested. To request a cancellation you just invoke the cancel method on the same pplx::cancellation_token_source instance you passed to your task. Typically you will call the cancel method from a different thread. Let’s take a look at this example:

void simple_cancellation()
{
    pplx::cancellation_token_source cts;

    auto t = pplx::task<void>([cts]()
    {
        while (!cts.get_token().is_canceled())
        {
            std::cout << "task is running" << std::endl;
            pplx::wait(90);
        }

        std::cout << "task cancelled" << std::endl;
    });

    pplx::wait(500);

    std::cout << "requesting task cancellation" << std::endl;
    cts.cancel();
    t.get();
}

Here is the output:

task is running
task is running
task is running
task is running
task is running
task is running
requesting task cancellation
task cancelled
Press any key to continue . . .

One noteworthy thing in the code above is that the pplx::cancellation_token_source variable is captured by value. This is possible because pplx::cancellation_token_source behaves like a smart pointer. It also guarantees thread safety. All of this makes it really easy to pass it around or capture it in lambda functions in this multi-threaded environment.

We now know how to cancel a single task. However, in real world it is very common to have continuations scheduled to run after a task completes. In most scenarios when cancelling a task you also want to cancel its continuations. You could capture the pplx::cancellation_token_source instance and do a check in each of the continuations but fortunately there is a better way. The ppxl::task::then() function has an overload that takes pplx::cancellation_token as a parameter. When using this overload the task will not run if the token passed to this function is cancelled. The following sample shows how this works:

void cancelling_continuations()
{
    pplx::cancellation_token_source cts;

    auto t = pplx::task<void>([cts]()
    {
        std::cout << "cancelling continuations" << std::endl;

        cts.cancel();
    })
    .then([]()
    {
        std::cout << "this will not run" << std::endl;
    }, cts.get_token());

    try
    {
        if (t.wait() == pplx::task_status::canceled)
        {
            std::cout << "task has been cancelled" << std::endl;
        }
        else
        {
            t.get(); // if task was not cancelled - get the result
        }
    }
    catch (const std::exception& e)
    {
        std::cout << "excption :" << e.what() << std::endl;
    }
}

Here is the output:

cancelling continuations
task has been cancelled
Press any key to continue . . .

In the code above the first task cancels the token. This results in canceling its continuation because we pass the cancellation token when scheduling the continuation. This is pretty straightforward. The important thing, however, is to also look at what happens on the main thread. In the previous example we called pplx::task::get(). We did this merely to block the main thread from exiting since it would kill all the outstanding threads including the one our task is running on. Here we call pplx::task::wait(). It blocks the main thread until the task reaches the terminal state and returns the status telling whether the task ran to completion or was cancelled. Note that if the task threw an exception this exception would be re-thrown from pplx::task::wait(). If this happens we need to handle this exception or the application will crash. Finally, for the completeness, we also have code that gets the result of the task (t.get()). It will never be executed in our case since we always cancel the task but in reality more often than not the task won’t be cancelled in which case you would want to get the result of the task.

There is a couple of questions I would like to drill into here:
– Why do we even need to use pplx::task::wait() – couldn’t we just use pplx::task::get()? The difference between pplx::task::wait() and pplx::task::get() is that pplx::task::get() will throw a pplx::task_canceled exception if a task has been cancelled whereas pplx::task::wait() does not throw for cancelled tasks. Therefore by using pplx::task::wait() you can avoid exceptions for cancelled tasks. This also may simplify your code when you handle exceptions in a different place than you would want to handle cancellation
– What happens if we don’t call pplx::task::wait()/pplx::task::get() and the task is cancelled? Sometimes we want to start an activity in the “fire-and-forget” manner and we don’t really care about the result so calling pplx::task::wait()/pplx::task::get() does not seem to be necessary. Unfortunately this won’t fly – if you call neither pplx::task::wait() nor pplx::task::get() on a task that has been cancelled a pplx::task_canceled exception will be eventually thrown which will crash your app – the same way as any other unobserved exception would do.

In the previous example we blocked the main thread to be able to handle the cancellation. This is not good – we use asynchronous programming to avoid blocking so it would be nice if we figured something out to not block the main thread. An alternative to handling a cancellation in the main thread is to handle cancellations in continuations. We can use the same pattern we used before to handle exceptions i.e. we can create a task based continuation in which we will call pplx::task:wait() to get the task status or pplx::task::get() and get a potential pplx::task_canceled exception. The important thing is that the continuation must not be scheduled using the overload of the pplx::then() function that takes the cancellation token. Otherwise the continuation wouldn’t run if the task was cancelled and the cancellation wouldn’t be handled resulting in an unobserved pplx::task_canceled exception that would crash your app. Here is an example of handling cancellation in a task based continuation:

void handling_cancellation_in_continuation()
{
    pplx::cancellation_token_source cts;

    auto t = pplx::task<void>([cts]()
    {
        std::cout << "cancelling continuations" << std::endl;

        cts.cancel();
    })
    .then([]()
    {
        std::cout << "this will not run" << std::endl;
    }, cts.get_token())
    .then([](pplx::task<void>previous_task)
    {
        try
        {
            if (previous_task.wait() == pplx::task_status::canceled)
            {
                std::cout << "task has been cancelled" << std::endl;
            }
            else
            {
                previous_task.get();
            }
        }
        catch (const std::exception& e)
        {
            std::cout << "exception " << e.what() << std::endl;
        }
    });

    t.get();
}

and the output:

cancelling continuations
task has been cancelled
Press any key to continue . . .

This is mostly it. There is one more topic related to cancellation – linked cancellation token sources but it is beyond the scope of this post. I believe that information included in this post should make it easy to understand how linked cancellation token sources work and how to use them if needed.

Arduino Pomodoro Timer

Some days are worse than other. On these bad days you don’t feel like working, people are constantly interrupting and, in general, not much is being done. When I have a day like this I use the Pomodoro technique to get back on track. The basic idea of the Pomdoro technique is that you try to focus solely on your work for 25 minutes minimizing all external and internal interruptions. You measure the time using a tomato shaped kitchen timer and hence the name – Pomodoro. Using the actual Pomodoro timer might seem like a good idea at first but in reality is not. The timer is ticking quite loudly which is annoying not only for you but also for people around. Using a software timer (I tried http://timemanage.me/) did not work for me either – other people can’t see it so it is impossible for them to know you are in the middle of your Pomodoro. On top of that you can’t check how much time is left without switching windows and losing focus which contradicts the very goal of Pomodoro (i.e. avoiding interruptions). This is why I decided to build my own Pomodorro timer. The requirements were simple – the timer had to be standalone so that I could check how much time is left with just one glimpse. To help prevent from external interruptions it also needed an indicator showing other people that I am busy. Meeting the first requirement was quite simple – a TM1637 based 4-digit LED display seemed good enough for the job. The second one was a bit harder. I got a little creative and bought a mood enhancing light (you should have seen my wife’s face when I showed her what I ordered) on Amazon. Now, that I got two most important parts the rest of the project was quite easy. I needed a few more parts like Arduino board (I went with Arduino Nano because of its size), a button (needed for restarting the timer), a color LED (to light my cube to indicate when I am busy) a small breadboard and a handful of resistors and jumper cables. This is the end result:

This project is quite easy but is really fun. You can try building a timer like this yourself. First connect the parts as shown on this picture:
PmodoroTimer_bb
Once the hardware part is done you need to upload the code to control the circuit. You can find the sketch in my ArduinoPomodoroTimer github repo. The most complicated part of the code is setting up the interrupt handler. I took it from my other project where I built a digital clock using an LCD display from an old Nokia phone. Take a look at this post for more details. Another complex piece is handling the LED display. Again, I wrote a very detailed post on this very subject a while ago so take a look at it to understand the code. The rest of the code is just about changing the color of the LED when the timer reaches 0 and reacting to button presses and is pretty straightforward.

Happy Pomodoring (is it even a word?)!