Author Archives: moozzyk

Automatic Reconnection in the Swift SignalR Client

As of version 0.7.0 the Swift SignalR Client supports automatic reconnection. This means that, if configured, the client will try to re-establish the connection to the server if the connection was lost. This post explains how this feature works, how to enable it and what configuration options are available.

Automatic reconnects

There are many scenarios where restoring an interrupted connection automatically is important. For instance, mobile applications very often have to be able to deal with unstable network and it’s crucial for these apps to be resilient to network issues. Conceptually the solution to the problem is simple – when the connection is lost a new connection needs to be started. In case of the Swift SignalR Client, this could always be implemented by the code consuming the client (i.e. on the application side). It turns out however, that in practice, implementing this logic is quite hard. Given that this is a common request and is the implementation is tricky it made sense to add support for automatic reconnection to the client. An important thing to note however is that the client does not offer anything more than just restoring the connection. In other words, the client will make a few attempts to restart the connection if it was stopped due to an error but will not do anything more than that. If reconnecting succeeds the server will treat the connection as a completely new connection and will assign it a new connection id. The client will also not receive messages it might have missed when it was disconnected. Anyone who used the non-Core version of SignalR can notice that this a big change to how reconnection worked in the non-Core version where, upon reconnecting, the server would recognize that the client has reconnected and resend missed messages. This functionality can no longer be implemented in the client for the Core version of SignalR because it requires cooperation from the server side (e.g. the server needs to buffer messages) and that logic does not exist in the Core version of SignalR.

Automatic reconnection is disabled by default. The main reason for this is backward compatibility – existing applications did not expect the connection to try to reconnect automatically so, they could break if the feature was enabled by default. Automatic reconnection requires also a bit of additional work and handle new lifecycle events.

Basics

The easiest way to enable automatic reconnection is to use the new .withAutoReconnect() method available on the HubConnectionBuilder class.

When automatic reconnection is enabled the application may receive two additional events:

  • connectionWillReconnect – invoked when the connection was lost
  • connectionDidReconnect – invoked when the connection was successfully restored

By default, the client will make up to four attempts to restore the connection. The first attempt will be made immediately after the connection was lost. The next three attempts will take place respectively 2, 10 and 30 seconds after the previous unsuccessful attempt. If all four attempts are unsuccessful the client will give up, close the connection and invoke the connectionDidClose event.

When the connection is being restored the client will not allow to invoke any method that tries to send data to the server.

Connection is now restartable

Adding support for automatic reconnection made the connection restartable. Before, once the connection was stopped it was necessary to create a new instance to be able to connect to the server again. This is no longer the case – the same instance can be used to restart connection to the server after it was stopped. This could be especially useful when handling background/foreground transitions.

Advanced Scenarios

The default reconnect configuration can be customized. It is possible to change the number of attempts as well as the time intervals between the attempts. The easiest way to do this is to create a new instance of the DefaultReconnectPolicy class with an array of retry intervals and pass this policy to the .withAutoReconnect() method. The number of retry intervals in the array tells the client how many reconnect attempts it should make, while the interval values indicate the time to wait between the attempts.

If the default reconnect policy is not flexible enough it is possible to go even further and create a custom reconnect policy by creating a class that conforms to the ReconnectPolicy protocol. This protocol has just one method – nextAttemptInterval that takes a RetryContext and returns the time interval telling the client when the next reconnect attempt should happen. The RetryContext instance passed to the nextAttempInterval method contains information about the current reconnect – the time when the reconnect was initiated, the number of failed attempts so far and the original error that triggered the reconnect. To stop further reconnect attempts the nextAttempInterval method should return DispatchTimeInterval.never. To put the policy to work the policy needs to be passed to the .withAutoReconnect() method when configuring the connection.

Backward Compatibility (a.k.a. Kill Switch)

As noted above, writing reconnect logic turned out to be quite tricky. It also required modifying existing code that is executed even when automatic reconnects are disabled. This created a risk of introducing issues into existing scenarios. In case of running into a bug like this it is possible to go back to the previous behavior by using the .withLegacyHttpConnection() method on the HubConnectionBuilder when creating a new hub connection.

Conclusion

These are pretty much all the details needed to be able to use automatic reconnection in the Swift SignalR Client. My hope is that automatic reconnection will make lives of developers much easier.

Swift Client for the Asp.NET Core version of SignalR – Part 2: Beyond the Basics

In the previous post we looked at some basic usage of the Swift SignalR Client. This was enough to get started but far from enough for any real-world application. In this post we will look at features offered by the client that allow handling more advanced scenarios.

Lifecycle hooks

One very important detail we glossed over in the previous post was related to starting the connection. While starting the connection seems to be as simple as invoking:

hubConnection.start()
view raw Start.swift hosted with ❤ by GitHub

it is not really the case. If you run the playground sample in one go you will see a lot of errors similar to:

2019-07-29T16:05:00.987Z error: Attempting to send data before connection has been started.

What’s going on here? The start() method is a not blocking call and establishing a connection to the server requires sending some HTTP requests, so takes much more time than just running code locally. As a result, the playground code continues to run and try to invoke hub methods while the client is still working in the background on setting up the connection. Another problem is that there is actually no guarantee that the connection will be ever successfully started (e.g. the provided URL can be incorrect, the network can be down, the server might be not responding etc.) but the start() method never returns whether the operation completed succcessfully. The solution to these problems is the HubConnectionDelegate protocol. It contains a few callbacks that allow the code that consumes the client be notified about the connection lifecycle events. The HubConnectionDelegate protocol looks like this:
public protocol HubConnectionDelegate: class {
func connectionDidOpen(hubConnection: HubConnection)
func connectionDidFailToOpen(error: Error)
func connectionDidClose(error: Error?)
}

The names of the callbacks should make their purpose quite clear but let’s go over them briefly:

  • connectionDidOpen(hubConnection: HubConnection)
    raised when the connection was started successfully. Once this event happens it is safe to invoke hub methods. The hubConnection passed to the callback is the newly started connection
  • connectionDidFailToOpen(error: Error) – raised when the connection could not be started successfully. The error contains the reason of the failure
  • connectionDidClose(error: Error?) – raised when the connection was closed. If the connection was closed due to an error the error argument will contain the reason of the failure. If the connection was closed gracefully (due to calling the stop() method) the error will be nil. Once the connection is closed trying invoking a hub method will result in an error

To set up your code to be notified about hub connection lifecycle events you need to create a class that conforms to the HubConnectionDelegate protocol and use the HubConnectionBuilder.withHubConnectionDelegate() method to register it. One important detail is that the client uses a weak reference to the delegate to prevent retain cycles. This puts the burden of maintaining the reference to the delegate on the user. If the reference is not maintained correctly the delegate might be released prematurely resulting in missing event notifications.
The example chat application shows the usage of the lifecycle events. It blocks/unblocks the UI based on the events raised by hub connection to prevent the user from sending messages when there is no connection to the server. The HubConnectionDelegate derived instance is stored in a class variable to ensure that the delegate will not be released before the connection is stopped.

HubConnectionBuilder

The HubConnectionBuilder is a helper class that contains a number of methods for configuring the connection:

  • withLogging – allows configuring logging. By default no logging will be configured and no logs will be written. There are three overloads of the withLogging method. The simplest overload takes just the minimum log level which can be one of:
    • .debug (= 4)
    • .info (= 3)
    • .warning (= 2)
    • .error (= 1)

    When the client is configured with this overload all log entries at the configured or higher log level will be written using the print function. The user can create more advanced loggers (e.g. a file logger) by creating a class conforming to the Logger protocol and registering it with one of the other withLogging overloads

  • withHubConnectionDelegate – configures a delegate that allows receiving connection lifecycle events (described above)
  • withHttpConnectionOptions – allows setting lower level configuration options (described below)
  • withHubProtocol – used to set the hub protocol that the client will use to communicate with the server. Not very useful at the moment given that currently the only supported hub protocol is the Json hub protocol which is also used by default (i.e. no additional configuration is required to use this protocol)

HttpConnectionOptions

The HttpConnectionOptions class contains lower level configuration options set using the HubConnectionBuilder.withHubConnectionOptions method. It allows configuring the following options:

  • accessTokenProvider – used to set a token provider factory. Each time the client makes an HTTP request (currently – because the client supports only the webSocket transport – this happens when sending the negotiate request and when opening a webSocket) the client will invoke the provided token factory and set the Authorization HTTP header to:
    Bearer {token-returned-by-factory}
  • skipNegotiation – by default the first step the client takes to establish a connection with a SignalR server is sending a negotiate request to get the capabilities of the server (e.g. supported transports), the connection id which identifies the connection on the server side and a redirection URL in case of Azure SignalR Service. However, the webSocket transport does not need a connection id (the connection is persistent) and if the user knows that the server supports the webSocket transport the negotiate request can be skipped saving one HTTP request and thus making starting the connection faster. The default value is false. Note: when connecting to Azure SignalR service this setting must be set to false regardless of the transport used by the client
  • headers – a dictionary containing HTTP headers that should be included in each HTTP request sent by the client
  • httpClientFactory – a factory that allow providing an alternative implementation of the HttpClient protocol. Currently used only by tests

Azure SignalR Service

When working with Azure SignalR Service the only requirement is that the HttpConnectionOptions.skipNegotiation is set to false. This is the default setting so typically no special configuration is required to make this scenario work.

Miscellaneous

Limits on the number of arguments

The invoke/send methods have strongly typed overloads that take up to 8 arguments. This should be plenty but in rare cases when this is not enough it is possible to drop to lower level primitives and use functions that operate on arrays of items that conform to the Encodable protocols. These functions work for any number of arguments and can be used as follows:

hubConnection.invoke(method: "Add", arguments: [2, 3], resultType: Int.self) { result, error in
if let error = error {
print("error: \(error)")
} else {
print("Add result: \(result!)")
}
}
view raw InvokeArray.swift hosted with ❤ by GitHub

Variable number of arguments

The SignalR server does not enforce that the same client method is always invoked with the same number of arguments. On the client side this rare scenario cannot be handled with the strongly typed .on methods. In addition -similarly to the scenarios described above – there is a limit of 8 parameters that the strongly typed .on callbacks support. Both scenarios can be handled by dropping to the lower level primitive which uses an ArgumentExtractor class instead of separate arguments. Here is an example:

hubConnection.on(method: "AddMessage", callback: { argumentExtractor in
let user = try argumentExtractor.getArgument(type: String.self)
var message = ""
if argumentExtractor.hasMoreArgs() {
message = try argumentExtractor.getArgument(type: String.self)
}
print(">>> \(user): \(message)")
})

These are pretty much all the knobs and buttons that the Swift SignalR Client currently offers. Knowing them allows using the client in the most effective way.

Swift Client for the Asp.NET Core version of SignalR – Part 1: Getting Started

SignalR-Client-Swift is a SignalR client for the Core version of SignalR for applications written in Swift. It’s been around for a while and, although the work is still in progress, it is stable and usable enough to use it in real apps. The project is an open source project hosted on GitHub and has received number of contributions from the community (e.g. including big features like support for Swift Package Manager). Unfortunately, so far, the documentation for this client has been between scarce and non-existent making it harder to adopt it. This and the next post aim to fix this problem.

Before looking diving into code let’s talk about the current state of affairs. As I mentioned, the work is far from finished and some features you can find in other clients are not currently supported. The following is the list of major SignalR features that are currently not implemented:

  • Long Polling and Server Sent Events transports
  • non-Json based hub protocols (e.g. Message Pack)
  • restartable connections
  • KeepAlive messages

Here is the more positive list of major features that are implemented:

  • webSockets transport
  • client and server hub method invocations (using Json hub protocol)
  • streaming methods
  • support for Azure SignalR Service
  • authentication with auth tokens

SignalR for ASP.Net Core is not backwards compatible with the previous version of SignalR. Hence, the Swift SignalR client will not work with the non-Core version of SignalR server.

Installation

The first step to use the client in a project is installation. Currently there are three ways to install Swift SignalR Client into your project:

CocoaPods

Add the following lines to your Podfile:

use_frameworks!
pod 'SwiftSignalRClient'
view raw CocoaPods hosted with ❤ by GitHub

Then run:
pod install

Swift Package Manager (SPM)

Add the following to your Package dependencies:

.package(url: "https://github.com/moozzyk/SignalR-Client-Swift", .upToNextMinor(from: "0.6.0")),

Then include "SignalRClient" in your target dependencies. For example:

.target(name: "MySwiftPackage", dependencies: ["SignalRClient"]),

Manually

Pull the code from the GitHub repo and configure SignalR client as an Embedded Framework.

Usage

Once the client has been successfully installed it is ready to use. The usage of the Swift SignalR Client does not differ much from other existing clients – you need to create a hub connection instance that you will use to connect and talk to the server. Note that you need to use the same instance of the client for the entire lifetime of your connection.
The easiest way to create a HubConnection instance is to use the HubConnectionBuilder class which contains a number of methods that allow configuring the connection to be created. For instance, creating a HubConnection instance with logging configured at the debug level would look like this:

let hubConnection = HubConnectionBuilder(url: URL(string: "http://localhost:5000/playground")!)
.withLogging(minLogLevel: .debug)
.build()

Creating a hub connection does not automatically start the connection. It just creates an instance that will be used to communicate with the server once the connection is started. This pattern makes it possible to register handlers for the client-side methods without risking missing invocations received between starting the connection and registering the handler. Handlers for the client-side methods are registered with the on method as follows:

hubConnection.on(method: "AddMessage") {(user: String, message: String) in
print(">>> \(user): \(message)")
}
view raw On.swift hosted with ❤ by GitHub

It is worth noting that types for the handler parameters must be specified and must be compatible with the types of values sent by the server (e.g. if the server invokes the method with a string the parameter type of the handler cannot be Int). The number of handler parameters should match the number of arguments used to invoke the client-side method from the server side.

After registering handlers it’s time to start the connection. It is as easy* as:

hubConnection.start()
view raw Start.swift hosted with ❤ by GitHub

From this point on, if the connection was started successfully, the handlers for the client-side methods will be invoked whenever the method was invoked on the server. Starting the connection allows also to invoke hub methods on the server side. (Trying to invoke a hub method on a non-started connection results in an error). SignalR supports two kinds of hub methods – regular and streaming. When invoking a regular hub method, the client may choose to be notified when the invocation has completed and receive the result of invocation (if the hub method returned any) or an error in case of an exception. Below are examples of such invocations:

// invoking a hub method and receiving a result
hubConnection.invoke(method: "Add", 2, 3, resultType: Int.self) { result, error in
if let error = error {
print("error: \(error)")
} else {
print("Add result: \(result!)")
}
}
// invoking a hub method that does not return a result
hubConnection.invoke(method: "Broadcast", "Playground user", "Sending a message") { error in
if let error = error {
print("error: \(error)")
} else {
print("Broadcast invocation completed without errors")
}
}
view raw Invoke.swift hosted with ❤ by GitHub

When invoking a hub method that returns a result providing the type of the result is mandatory and this type has to be compatible with the type of the value returned by the hub method. Also, there is no distinction between local and remote handlers – i.e. the completion handler will be called with an error not only when the method on the server side fails but also when initiating the invocation fails (e.g. when trying to invoke a method when the connection is not running).

Hub methods can also be invoked in a fire-and-forget manner. When invoking a hub method in this fashion the client will not be notified when the invocation has completed and will not receive any further events related to this method – be it a result or an error. The code below shows how to invoke a hub method in a fire-and-forget manner:

hubConnection.send(method: "Broadcast", "Playground user", "Testing send") { error in
if let error = error {
print("Send failed: \(error)")
}
}
view raw Send.swift hosted with ❤ by GitHub

Note, that the send method still takes a callback that allows handling errors but this callback will be called only for local errors – i.e. errors that occurred when sending data to the server.

SignalR streaming hub methods return a (possibly infinite) stream of items. Each time the client receives a new stream item a user provided callback will be invoked with the received value.
When a streaming method completes executing a completion callback will be invoked (except for this bug which I found writing this post). The client method that invokes streaming hub methods returns a stream handle. This handle can be used to cancel the streaming hub method. The following code snippet illustrates how to invoke and cancel a streaming hub method:

let streamHandle = hubConnection.stream(method: "StreamNumbers", 1, 10000, itemType: Int.self,
streamItemReceived: { item in print(">>> \(item!)") }) { error in
print("Stream closed.")
if let error = error {
print("Error: \(error)")
}
}
DispatchQueue.main.asyncAfter(deadline: .now() + .seconds(2)) {
hubConnection.cancelStreamInvocation(streamHandle: streamHandle) { error in
print("Canceling stream invocation failed: \(error)")
}
}
view raw Stream.swift hosted with ❤ by GitHub

If you no longer want to receive notifications from the server or invoke hub methods you can disconnect from the server with:

hubConnection.stop()
view raw Stop.swift hosted with ❤ by GitHub

One final note about types of arguments and results. The types of all the values sent to the server must conform to the Encodable protocol. The types for the values returned from the server must conform to the Decodable protocol. The most common types in Swift already conform to the Codable protocol (which means that they conform to both the Encodable and the Decodable protocols) and when creating custom structs/classes it is easy to make them conform to the Codable protocol as long as all the member variables already conform to the Codable protocol.

These are the basics of the SignalR Swift Client. The project repo contains additional resources in form of example applications for macOS and iOS. I also created a Swift playground which contains all code snippets published in this post. In the next post we will look at the connection lifecycle events, available configuration options and more advanced scenarios.

* – it is actually not entirely true but we will return to it in the second post†

Hello, OCaml!

I am a huge fan of the Advent of Code. I eagerly wait for it every year and once it starts I try to solve the problems the same day they appear. The nice thing about the problems is that they are initially relatively simple and get more difficult over time (and if you ever get stuck there is a subreddit for each problem that you can use to unblock yourself). This makes it a fantastic opportunity to try new languages. (Note, I intentionally said “try” and not “learn” because the vast majority of the problems can be solved in less than 100 lines of code and almost never require advanced data structures and/or language features). And this is what I do – each year I pick a language I have never used before and solve all the problems using this language. Two years ago – the first edition of the Advent of Code – I tried Scala, last year I tried Go and this year I ended up picking OCaml (with Rust and D being other contenders). One thing I decided to do differently this year however, was to write down my observations and things I struggled with to share them, hoping that it will help people trying to learn OCaml.

Preparing Development Environment

Before I could start solving the problems I needed to setup my dev environment and figure out how to build and run programs written in OCaml. I used Visual Studio Code as my editor because I was sure it would have an extension for OCaml (I used OCaml & Reason IDE). To build my programs I ended up using ocamlbuild although probably it was an overkill – I never had more than one file to compile and used at most one additional library (Str). With my dev environment set up I was ready to start solving Advent of Code problems and learning OCaml the hard (i.e. trial-and-error) way.

Common errors

When you try to pick a new language, you will initially make a lot of simple, syntactic mistakes. I think this was one of the biggest barriers to me at the beginning. I would try to compile my program and the compilation would fail with an error I could only stare at. Understanding the error would initially take me a lot of time and experimenting (like commenting out code etc.). After a few days I figured what errors my most common mistakes resulted in and things got much easier, but I would still occasionally encounter an error which took a relatively long time to resolve. (Note, some of the mistakes could probably have been avoided if I had read more on the language before I started coding but this is not the way I learn – I prefer reading just enough to be able to do simple things and then figure out things as I go).
Before I dive deeper into errors I encountered I would like to give a few hints that can help in locating and understanding the error:

  • The location of the error contains the line and the column where the error occurs. The column can be very helpful – especially when you invoke a function with multiple parameters or inline a function invocation
  • Oftentimes the mistake is not in the line the error message points to. If you can’t find anything wrong with the line the error message points to check the line(s) the function is invoked from
  • Learn how to read function signatures (e.g. int -> int -> int) to easier understand errors caused by passing values of incorrect types. See the Types of Functions in the OCaml tutorial.

During my adventure with OCaml I compiled a list of errors I encountered.

Syntax error

This was initially the most common error I saw. It can have many causes and I am sure that the list below is not exhaustive. Most common causes:

  • missing in after variable declaration:
    let sum a b =
    let s = a + b
    print_int
    (* Error: Syntax error *)
    let sum a b =
    let s = a + b in
    print_int s
    (* val sum : int -> int -> unit = <fun> *)
    view raw missing_in.ml hosted with ❤ by GitHub
  • using => instead of -> (likely specific to developers who mostly use C# or JavaScript/TypeScript)
  • using reserved words as variable names (I fell a few times for val and match I wanted to use respectively for a variable storing a temporary value and a result of regular expression match)
  • missing -> in pattern matching (the exact error is Syntax error: pattern expected

Unbound value

This error means that the function you are trying to call cannot be found. Most common causes:

  • You have a typo in the function name
  • Your recursive function is not marked rec:
    let factorial n =
    if n = 0 then 1
    else n * factorial(n - 1)
    (* Error: Unbound value factorial *)
    let rec factorial n =
    if n = 0 then 1
    else n * factorial(n - 1)
    (* val factorial : int -> int = <fun> *)
    view raw missing_rec.ml hosted with ❤ by GitHub

This function has type X It is applied to too many arguments; maybe you forgot a `;'.

Similarly to Syntax error there can be multiple causes for this error:

  • You actually forgot a ; to separate your statements:
    print_string "a"
    print_string "b"
    (* Error: This function has type string -> unit
    It is applied to too many arguments; maybe you forgot a `;'. *)
    print_string "a";
    print_string "b"
    (* Result: ab *)
    view raw missing_semicolon.ml hosted with ❤ by GitHub
  • You passed too many arguments to a function:
    let sum list = List.fold_left (fun v a -> a + v) 0 list
    sum [1;2;3] "a"
    (* Error: This function has type int list -> int
    It is applied to too many arguments; maybe you forgot a `;'. *)
    sum [1;2;3]
    (* Result: 6 *)
    view raw too_many_params.ml hosted with ❤ by GitHub
  • You forgot parenthesis when inlining a function (similar to a previous case but sometimes harder to notice):
    print_int max 4 5
    (* Error: This function has type int -> unit
    It is applied to too many arguments; maybe you forgot a `;'. *)
    print_int (max 4 5)
    (* Result: 5 *)

WTF errors

The following errors fall into one of the categories above but can be very hard to spot a beginner (like I was):

  • Arithmetical operations, string or list concatenation must be in parentheses when used as an argument to a function :
    print_int 2 + 3
    (* Error: This expression has type unit but an expression was expected of type
    int *)
    print_int (2 + 3)
    (* Result: 5 *)
    print_string "a"^"b"
    (* Error: This expression has type unit but an expression was expected of type
    string *)
    print_string ("a"^"b")
    (* Result: ab *)
  • Negative values also should use parentheses:
    print_int -5
    (* Error: This expression has type int -> unit
    but an expression was expected of type int *)
    print_int (-5)
    (* Result: -5 *)
  • Sometimes the error points to a line that does not seem to be the cause of the problem. For instance in the following example I did not convert the argument of the sqrt function to the float type but the error points to a totally different line:
    let is_prime n =
    let rec is_prime_aux n div =
    if div > int_of_float (sqrt n) then true
    else if n mod div = 0 then false
    else is_prime_aux n (div + 1)
    in
    is_prime_aux n 2
    (* Error: This expression has type float but an expression was expected of type
    int
    Points to `n` in `if n mod div...` *)
    let is_prime n =
    let rec is_prime_aux n div =
    if div > int_of_float(sqrt (float_of_int n)) then true
    else if n mod div = 0 then false
    else is_prime_aux n (div + 1)
    in
    is_prime_aux n 2
    view raw float_conversion.ml hosted with ❤ by GitHub
  • Errors caused by partial application triggered by unintentionally passing fewer parameters that the function requires.

Observations

If you don’t want a biased opinion about OCaml you can stop reading now 🙂

Overall, I did not enjoy OCaml as a programming language. Initially, I struggled with errors that did not have much meaning to me. Once I got past this phase I found that I was not very productive – even conceptually simple problems required too much code for my taste. Maybe part of it was me learning the language but there were other people doing Advent of Code in OCaml and I found their solutions rarely required less code than mine. I also thought that it might be because I don’t use functional programming in my day-to-day work (except for quasi-functional features of C# like LINQ) and can’t switch to the functional paradigm but I briefly looked at my solutions in Scala from 2015 and they are generally much more compact.

Because I wanted to learn the language and not the libraries I wanted to get away as much as possible with just the basic language features (i.e. without using external libraries). It turned out to be hard. One of the most basic operation is writing a formatted string. OCaml offers a number of print_* functions. Unfortunately, these functions are very basic and even writing a number followed by a new line requires two print statements. Printing a formatted string with these functions was so cumbersome that I eventually decided to use the Printf module.

Another thing that baffled me at the very beginning was reading file contents. Most Advent of Code problems require reading the puzzle input from a file and I could not find any simple method to read lines from a file. It took me more time than I wanted to spend on this basic problem. All solutions seemed overly complicated. I eventually found this function on stackoverflow:

let read_lines name : string list =
let ic = open_in name in
let try_read () =
try Some (input_line ic) with End_of_file -> None in
let rec loop acc = match try_read () with
| Some s -> loop (s :: acc)
| None -> close_in ic; List.rev acc in
loop []
view raw reading_lines.ml hosted with ❤ by GitHub

I have to admit that being a novice I initially did not fully understand how it worked but hey, it did work. The other alternatives I found were either even more complicated or required understanding concepts I did not event want to learn at the time (e.g. channels). All in all, I have to say there is a heck of complexity to achieve a very basic task that in many languages is just a single, self-explanatory line of code – e.g. File.ReadAllLines("myFile.txt").

I was also annoyed but by the lack of functions for simple string processing. To treat a string to as a sequence of characters you need to do something like this (again, I am not the author of this function):

let explode s =
let rec exp i l =
if i < 0 then l else exp (i - 1) (s.[i] :: l) in
exp (String.length s - 1) []
view raw string_to_chars.ml hosted with ❤ by GitHub

In reality, it’s hard to do string processing without using the Str module (btw. compare the name with the String module that contains basic string operations and is part of the standard library and tell me how you would not be confused which one is which) which supports regular expressions.

Speaking of regular expressions I found them weird to use. The first weird thing was that you could not just get all matched groups (e.g. as a list). Rather, you need to enumerate them one by one without knowing how many groups where matched (so you potentially enumerate until you hit an error). The other weird thing was that it seems that matched groups (or maybe some state that allows calculating them) are stored in the library and are keyed by the original string that was processed. Each time you want to get a matched group you need to provide this original string.

API inconsistencies are also irritating. For instance, both List and Hashtbl have the fold_left function. However, while List takes the function, the initial value and the list as parameters, Hashtbl takes the function, the hash table and the initial value (i.e. the initial value and the container are swapped comparing to List). It seems like a small thing, but I hit this many, many times.

Similarly, if you have multiple statements you need to separate them with a semicolon (;). However, if you use multiple statements in the then or else clause they also need to be wrapped with begin/end (or parentheses). On the other hand, you don’t need begin/end if you have multiple statements inside a loop (yes, I know, I should use recursion but sometimes using a loop is, you know, just simpler).

Runtime errors are a nightmare. You don’t get any details about the error except for the message. This might work if an exception is thrown from your own code but how you are supposed to find the bug effectively if all you get is:
Fatal error: exception Invalid_argument("index out of bounds")?

The last thing that I was surprised by was how scarce the documentation for OCaml is. When I decided to use OCaml to solve Advent of Code problems I knew it was not one of the mainstream programming languages, but I did not consider it completely niche. As soon as I started I realized that there is only a very limited number of resources at my disposal. Ironically, one of the best turned out to be the library reference on the https://caml.inria.fr website whose main page says:
This site is updated infrequently. For up-to-date information, please visit the new OCaml website at ocaml.org.
The library reference looks a bit raw and dated (e.g. see: https://caml.inria.fr/pub/docs/manual-ocaml/libref/String.html) and at the beginning I had a hard time digesting the information it provided but once I got more familiar with OCaml I would visit it all the time (yes, 50% of my visits were to check if the initial value for the fold_left function should go before or after the container).

To sum up – the next time I start a new project and have freedom to choose the language for the project I don’t think OCaml will be on the list. I am glad I tried it but I hoped for a more pleasant ride.

Coincidentally, HackerRank released their 2018 Developer Skill Report recently and OCaml was one of only two languages with negative sentiment among developers of all age groups. I will just say that the other language was Perl.

P.S. My solutions to Advent of Code 2017 can be found on github.

Using Raspberry Pi to flash ESP8266

Necessity is mother of invention. Even though in this case there was not a lot of invention, there was necessity. I wanted to play with the ESP8266 module and I wanted it badly. Unfortunately, I could not find any FTDI in my drawers, so I had to find a different way of talking to my ESP8266. I quickly concluded that I could use either Arduino or Raspberry Pi. I decided to start with Arduino but for whatever reason could not make it work. A little desperate, I decided to try Raspberry PI. This, thanks to the information I found on the Internet, got me much further. I found a few articles scattered over the web and after combining various pieces of information I was able not only to talk to my ESP8266 but also reprogram it. I decided to write this post so that all the information I found is in one place. Let’s get started.

The idea is to connect an ESP8266 module to Raspberry Pi’s GPIO serial pins and then use a serial communication terminal like minicom or screen to talk to the module. Before we can do this, we need to reconfigure our Raspberry Pi so that the OS is not using the serial interface. The first step is to disable serial logins. The easiest way to achieve this is to use raspi-config. Start the the raspi-config with sudo raspi-config, go to Interfacing Options and then to Serial. You will be asked two questions – answer them as follows:

  • Would you like a login shell to be accessible over serial?
    Select: No
  • Would you like the serial port hardware to be enabled?
    Select: Yes

The second thing to do is to switch off sending bootup info to serial by removing references to ttyAMA0 in the /boot/cmdline.txt file – e.g.

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=PARTUUID=9815a293-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

becomes:

dwc_otg.lpm_enable=0 console=tty1 root=PARTUUID=9815a293-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Finally, you need to restart your Raspberry Pi for changes to take effect

sudo reboot now

Now, that we prepared our Raspberry Pi we need to wire the ESP8266 module. You need to power down Raspberry PI and connect your ESP8266 module as per the diagram below.

RPI_ESP8266_normal_mode


ESP8266 Raspberry Pi
GND       GND
3.3V      3.3V
RXD       UART_TXD
TXD       UART_RXD
CH_PD     3.3V (via a pull down resistor)

After wiring the ESP8266 module to the Raspberry Pi has been completed we can boot the Pi and connect to the ESP8266 using screen (I am using screen because it is installed by default but you can use minicom if you prefer):

screen /dev/ttyAMA0 115200

To see if we connected successfully let’s send some commands to the module (note: when using screen you need to press Enter and CTRL+J to send the command).

Check if AT system works (AT):

AT
OK

Display the version (AT+GMR)

AT+GMR
AT version:1.2.0.0(Jul  1 2016 20:04:45)
SDK version:1.5.4.1(39cb9a32)
Ai-Thinker Technology Co. Ltd.
Dec  2 2016 14:21:16
OK

Connect to WiFi (a series of commands)

AT+CWMODE_CUR=1
OK

AT+CWJAP_CUR="ssid","pwd"
WIFI CONNECTED
WIFI GOT IP
OK

AT+CWJAP_CUR?
+CWJAP_CUR:"ssid","18:a6:f7:23:9e:50",6,-52
OK

AT+CIFSR
+CIFSR:STAIP,"192.168.0.110"
+CIFSR:STAMAC,"5c:cf:7f:36:cd:31"

The full list of AT commands can be found on the espressif’s web site in the documents section – look for ESP8266 AT Instruction Set. Note that some commands from the document may not work if you are using an older version of the firmware.

We are able to communicate with our ESP8266. Now, let’s try updating the firmware. First, we need to power down the PI again and connect the GPIO0 pin of the ESP8266 to the PI’s ground pin. This will make the module run in the flash mode.

RPI_ESP8266_flash_mode

The next step is to install the esptool which we will use to transfer firmware files to the module:

sudo apt-get update
sudo apt-get install python
sudo apt-get install python-pip
sudo pip install esptool

Finally, we need a firmware that we will flash to the device. We can get a new firmware by cloning the espressif’s ESP8266_NONOS_SDK  repo:

git clone https://github.com/espressif/ESP8266_NONOS_SDK

After the repo has been cloned we need to go to the bin folder and run the following command:


esptool.py --port /dev/ttyAMA0 --baud 115200  write_flash --flash_freq 40m --flash_mode qio 0x0000 boot_v1.7.bin 0x1000 at/512+512/user1.1024.new.2.bin 0x7E000 blank.bin

The output of the command should look like this:

esptool.py v2.0.1
Connecting...
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 1MB
Flash params set to 0x0020
Compressed 4080 bytes to 2936...
Wrote 4080 bytes (2936 compressed) at 0x00000000 in 0.3 seconds (effective 121.5 kbit/s)...
Hash of data verified.
Compressed 427060 bytes to 305755...
Wrote 427060 bytes (305755 compressed) at 0x00001000 in 27.9 seconds (effective 122.3 kbit/s)...
Hash of data verified.
Compressed 4096 bytes to 26...
Wrote 4096 bytes (26 compressed) at 0x0007e000 in 0.0 seconds (effective 4538.5 kbit/s)...
Hash of data verified.

Once the process is complete we can power down the Raspberry Pi and disconnect the ESP8266 GPIO0 pin from GND to make it run again in the normal mode. Now, we can start the PI, and dump the ESP8266 firmware version with the AT+GMR command:


AT+GMR
AT version:1.4.0.0(May  5 2017 16:10:59)
SDK version:2.1.0(116b762)
compile time:May  5 2017 16:37:48
OK

This is a much newer version than what we originally had so we were able to flash our ESP8266 successfully. Yay!

Acknowledgements:

The following articles were instrumental in writing this post:

 

 

The SignalR for ASP.NET Core JavaScript Client, Part 2 – Outside the Browser

Last time we looked at using the ASP.NET Core SignalR TypeScript/JavaScript client in the browser. I mentioned, however, that the new client no longer has dependencies that prevent from using it outside the browser. So, today we will try taking the client outside the browser and use it in a NodeJS application. We will add a NodeJS client for the SignalR Chat service we created last time. Initially we will write the client in JavaScript and then we will convert it to TypeScript.

Let’s start from creating a new folder in the SignalRChat repo and adding a new node project:

mkdir SignalRChatNode
cd SignalRChatNode
npm init

We will call the application signarlchatnode and we will leave all other options set to default values. (6425ec1)

Our application will read messages typed by the user and send them to the server. To handle user input we will use node’s readline module. To see that things, work, let’s just add code to prompts the user for the name and displays it in the console. We will use it a starting point of our application (34bc493).

const readline = require('readline');
let rl = readline.createInterface(process.stdin, process.stdout)

rl.question('Enter your name: ', name => {
console.log(name);
  rl.close();
});

To communicate with the SignalR server we need to add the SignalR JavaScript client to the project using the following command (7875c07):

npm install @aspnet/signalr-client --save

We can now try starting the connection like this (3228a10):

const readline = require('readline');
const signalR = require('@aspnet/signalr-client');

let rl = readline.createInterface(process.stdin, process.stdout);

rl.question('Enter your name: ', name => {
  console.log(name);

  let connection = new signalR.HubConnection('http://localhost:5000/chat');
  connection.start()
  .catch(error => {
    console.error(error);
    rl.close();
  });
});

The code looks good but if you try running it, it will immediately fail with the following error:

Error: Failed to start the connection. ReferenceError: XMLHttpRequest is not defined
ReferenceError: XMLHttpRequest is not defined

What happened? The new JavaScript client no longer depends on the browser but still uses standard libraries like XmlHttpRequest or WebSocket to communicate with the server. If these libraries are not provided the client will fail. Fortunately, the required functionality can be easily polyfilled in the NodeJS environment. For now, we will just stick the polyfills on the global object. It’s not beautiful by any means but will do the trick. We are discussing how to make it better in the future but at the moment this is the way to go.

Depending on the features of SignalR you plan to use you will need to provide appropriate polyfills. Currently the absolute minimum is XmlHttpRequest. SignalR client uses it to send the initial OPTIONS HTTP request which initializes the connection on the server side and for the long polling transport. So, if use the long polling transport only, XmlHttpRequest is the only polyfill you will need to provide . If you want to use the WebSockets transport you will need a WebSocket polyfill in addition to XmlHttpRequest. (We are thinking about skipping sending the OPTIONS request for WebSockets. If this is implemented you will not need the XmlHttpRequest polyfill when using the WebSockets transport.) For ServerSentEvents transport you will need an EventSource polyfill. Finally, if you happen to use binary protocols (e.g. MessagePack) over the ServerSentEvent transport you will need polyfills for atob/btoa functions. For simplicity, we will use the WebSocket transport in our application so we will add only polyfills for XmlHttpRequest and WebSockets:

npm install websocket xmlhttprequest --save

and make them available globally via:

XMLHttpRequest = require('xmlhttprequest').XMLHttpRequest;
WebSocket = require('websocket').w3cwebsocket;

If we run the code now we will see something like this:

moozzyk:~/source/SignalRChat/SignalRChatNode$ node index.js
Enter your name: moozzyk
moozzyk
Information: WebSocket connected to ws://localhost:5000/chat?id=0d015ce4-3a78-4313-9343-cb6183a5e8ea
Information: Using HubProtocol 'json'.

which tells us that the client was able to connect successfully to the server. (946f85d)

Now, we need to add some code to handle user input and interact with the server and our Node SignalR Chat client is ready. (I admit that the user interface is not very robust but should be enough for the purpose of this post). You can now talk to browser clients from your node client and vice versa (0f7f71f):

Screen Shot 2017-09-30 at 6.57.14 PM

Now let’s convert our client to TypeScript. We will start from creating a new TypeScript project with tsc --init. In the generated tsconfig.json file we will change the target to es6. We will also add an empty index.ts file and delete the existing index.js file (we will no longer need the index.js file since we will now be generating one by compiling the newly created index.ts). (b83cf92) If you now run tsc you should see an empty index.js file created as a result of compiling the index.ts file.  The last thing to do is to actually convert our JavaScript code to TypeScript. We could just translate it one-to-one but we can do a little better. TypeScript supports async/await which makes writing asynchronous code much easier. Since many of SignalR client methods return Promises we can just await these calls instead of using .then/.catch functions. Here is how our node SignalRChat client written in TypeScript looks like (2a6d0e9):

import * as readline from "readline"
import * as signalR from "@aspnet/signalr-client"

(<any>global).XMLHttpRequest = require("xmlhttprequest").XMLHttpRequest;
(<any>global).WebSocket = require("websocket").w3cwebsocket;

let rl = readline.createInterface(process.stdin, process.stdout);

rl.question("Enter your name: ", async name => {
  console.log(name);
  let connection = new signalR.HubConnection("http://localhost:5000/chat");

  connection.on("broadcastMessage", (name, message) => {
    console.log(`${name}: ${message}`);
    rl.prompt(true);
  });

  try {
    await connection.start();
    rl.prompt();

    rl.on("line", async input => {
      if (input === "!q") {
        console.log("Stopping connection...");
        connection.stop();
        rl.close();
        return;
      }
      await connection.send("send", name, input);
    });
  }
  catch (error) {
    console.error(error);
    rl.close();
  }
});

You can run it by executing the following commands:
tsc
node index.js

Today we learned how to use the ASP.NET Core SignalR client in the NodeJS environment. We created a small node JavaScript application that was able to communicate with browser clients which. Finally, we converted the JavaScript code to TypeScript and learn a little bit about the TypeScript’s async/await feature.

The SignalR for ASP.NET Core JavaScript Client, Part 1 – Web Applications

The first official release of SignalR for ASP.NET Core – alpha1 – was just released. In this release, all SignalR components were rewritten to make SignalR simpler, easier to use and more reliable.

The SignalR JavaScript client has always been a fundamental part of SignalR. Unfortunately, it has a few limitations which made it hard to extend or use outside the browser. The rewrite allowed to introduce changes which allow to take the client outside the browser (no more dependency on jQuery, YAY!) and open new scenarios. And this is what this blog post will focus on. I split the post to two parts. In the first part I will show how to use the client in a web application from both JavaScript and TypeScript. In the second, part we will look at NodeJS.

The plan for this part is to recreate the chat application from the tutorial on the previous version of SignalR and then to convert it to use the new SignalR Server and JavaScript client. The sample is simple enough to allow us to focus on SignalR aspects rather than on application intricacies. As a bonus, we will see what the experience of porting an application from the previous version of SignalR is. I created a github repo for the application where each commit is a step described in this post. I will refer to particular commits from this post to show changes for a given step.

Setting up the Server

Let’s start from creating an empty ASP.NET Core application. We can do that from command line by running the dotnet new web command. (See this step on github).

Once the application is created we can start the server with dotnet run and make sure it works by navigating to http://localhost:5000 from a browser.

After we ensured that the application runs we can add SignalR server components. First, we need to add a reference to the SignalR package to the SignalRChat.csproj file (See this step on github).

Now we can add the Chat Hub class – we will just copy the code from tutorial and tweak a few things. This is how the hub class looks after the changes:

using System;
using Microsoft.AspNetCore.SignalR;
namespace SignalRChat
{
    public class ChatHub : Hub
    {
        public void Send(string name, string message)
        {
            // Call the broadcastMessage method to update clients.
            Clients.All.InvokeAsync("broadcastMessage", name, message);
        }
    }
}

The changes we made were only cosmetic – we removed the reference to the System.Web namespace, added 'Core' to the Microsoft.AspNet.SignalR so that it reads Microsoft.AspNetCore.SignalR. We also changed how we invoke the client-side method by passing the method name as the first parameter to the InvokeAsync call. (See this step on github).

Now that we created a hub we need to configure the application to be aware of SignalR and to forward SignalR related messages to our hub. It’s as easy as calling AddSignalR extension method in the ConfigureServices method of our Startup class and mapping the hub with the UseSignalR method. We will also add the static files middleware which will be responsible for serving static files. The Startup class should look like this:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddSignalR();
    } 

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseFileServer();

        app.UseSignalR(routes =>
        {
            routes.MapHub<ChatHub>("chat");
        });
    }
}

(See this step on github).

And this is all the work we had to do create a functional SignalR chat server. Now we can focus on the client side.

The JavaScript Client

In the new version of SignalR the JavaScript client is distributed using npm. The npm module contains a version of the client that can be just included in a web page using the tag, as well as, typings and modules that can be consumed from TypeScript. To get the client to your machine you need to install npm if you haven’t already and run:

npm install @aspnet/signalr-client

The client will be installed in the node_modules folder and you can find the necessary files to include in the node_modules/@aspnet/signalr-client/dist/browser folder. You may wonder why there are so many files in this folder and what purpose they serve. Let’s go over them then and explain.

First, you will find that there are two sets of files – files that contain ES5 in the names and files that do not contain ES5 in the names. SignalR JavaScript uses ES6 (a.k.a EcmaScript 2015) features like Promises or arrow functions. Not all browsers however, support ES6 (looking at you Internet Explorer). The files without ES5 in the names are meant to be used in browsers that support ES6. The files that contain ES5 in the names are the ES6 files transpiled to ES5. They are ES5 compatible and include all required dependencies. The downside of the ES5 files is that they are much bigger than ES6 files.

Another interesting set of files are files containing msgpackprotocol in the name. The new version of SignalR supports custom hub protocols – including binary protocols – and has built-in support for a binary protocol based on MessagePack. The JavaScript implementation of the MessagePack based hub protocol (using the msgpack5) turned out to be quite big so we moved it to a separate file. This way you can include the MessagePack hub protocol only if you want to use it and will not pay the price if you don’t care.

You will also find that each file has a min counterpart. These are just minified versions of the corresponding files. You will want to use the minified versions in production but debugging is much easier with non-minified files so you may want to use non-minified versions during development.

Finally, there is also the third-party-notices.txt file. These are notices for the msgpack5 library and its dependencies used in the MessagePack hub protocol implementation.

Using the SignalR JavaScript Client from JavaScript

Now, that we know a little bit about the JavaScript client let’s update our application to use it.

First, let’s copy all the files from the node_modules/@aspnet/signalr-client/dist/browser folder to a new ​scritps/signalr folder under the wwwroot. (See this step on github).

After the files are copied, let’s create the index.html file in the wwwroot folder and paste the contents of the html file from the tutorial. (See this step on github).

If you try to run the application at this point it will not work. The index.html has references to files like the jQuery library or the old SignalR client which don’t exist. Let’s fix that. Note that even though jQuery is no longer required to the new SignalR client I will continue to use it to minimize the number of changes I need to make. All in all this is not a tutorial on how to remove jQuery from your app so let’s not get sidetracked. Let’s start from sorting out the scripts situation. For jQuery, I will replace the link with the one to the jQuery CDN. For SignalR, I will replace the link to the signalR-2.2.1.min.js file with signalR-client-1.0.0-alpha1.js (feel free to use the ES5 version if you are using a browser that don’t support ES6 features) and remove the link to hubs since hub proxies are currently not supported. (See this step on github (github trick – notice that the link ends with ?w=1 – try removing it and see what happens. Very useful when reviewing some PRs)).

Now we can finally fix the code. Fortunately, this is not a lot of changes:

  • Instead of using proxies we will just create a new HubConnection
  • To register the callback for the client side broadcastMessage method we will use the on function
  • We will replace the done method used by jQuery deferreds to the then used by ES6 promises
  • We will invoke hub methods with the invoke function

(See this step on github).

That’s pretty much it. If you run the application now you should be able to send and receive messages.

Using the JavaScript Client from TypeScript

We now know how to use the new JavaScript SignalR client from JavaScript code. The SignalR client module contains also all necessary bits that make it possible to be consumed from TypeScript. To see how it works let’s take our chat application a bit further and convert it TypeScript.

First, make sure that you have a recent TypeScript compiler installed – run tsc --version from command line. If running the command fails or you have an older version installed install the latest one using this command:

npm install typescript -g

After installing or updating the typescript compiler we will initialize a new project by running

tsc --init

in the project folder. This will create a tsconfig.json file which will look like this:

{
  "compilerOptions": {
    "target": "es6",
    "module": "commonjs",
    "strict": true,
    "noImplicitAny": true
  }
}

after performing some cleanup. We will also add a new chat.ts file which we will leave empty for now. If you run the tsc command from project root you should see an almost empty chat.js file generated from your chat.ts file. (See this step on github).

Because we are using TypeScript and will bring dependencies using npm we will no longer need JavaScript files for the browser so let’s delete them. (See this step on github).

To be able to add and restore dependencies the client will need, let’s create a package.json file by executint the npm init command. We will leave default values for almost all settings except for the project name which needs to be lowercase.

PS C:\source\SignalRChat\SignalRChat> npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (SignalRChat) signalrchat
version: (1.0.0)
description:
entry point: (chat.js)
test command:
git repository:
keywords:
author:
license: (ISC)
About to write to C:\source\SignalRChat\SignalRChat\package.json:

{
"name": "signalrchat",
"version": "1.0.0",
"description": "",
"main": "chat.js",
"dependencies": {},
"devDependencies": {},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC"
}

Is this ok? (yes)
PS C:\source\SignalRChat\SignalRChat>

Now let’s add our dependencies – signalr-client, jquery and jquery typings (they enable using jquery from TypeScript). We will use the --save-dev option to save the dependencies as dev dependencies in the package.json file.

npm install @aspnet/signalr-client --save-dev
npm install jquery --save-dev
npm install @types/jquery --save-dev

We also need to install browserify – a tool which we will use to create the final script to be used by the browser:

npm install -g browserify

(See this step on github).

We can now start working on the code. First, we need to import the dependencies we are going to use. We can do that by adding the following two lines at the top of our chat.ts file:

import * as signalR from "@aspnet/signalr-client"
import * as $ from "jquery

Now we can move the script from our .html file to the .ts file. If you do that and play a little bit with the code you will notice that intellisense now tells you about class members and function parameters and if you press F12 (in Visual Studio Code) it will take you to the function header. Another thing, you will see is an error on line 5.  This TypeScript telling you that there is a type mismatch for the parameter passed to the jQuery val() function – the prompt() function can return null which is not a valid input for the val() function.

VSCodeSignalR

In our case we know that prompt will return string so we will just cast the result to string to suppress the error.

Since we moved the function to the .ts file we can now remove all the JavaScript code from our index.html file. We can also remove all the tags since we no longer depend on them to bring dependencies (we also already deleted the scripts). (See this step on github).

Let’s compile our chat.ts file now by running tsc command. If you look at the generated chat.js file you will notice that it looks pretty much the same as the source chat.ts file with some additional lines at the top. You will also notice that it does not have the required dependencies (i.e. signalr-client and jquery). This is where browserify comes into play. We will use browserify to generate the final version of the file with all the dependencies. Let’s run the following command (you may need to create the wwwroot/scripts folder if one does not exist) from the project folder:

browserify .\chat.js -o .\wwwroot\scripts\chat.js

Take a look at the chat.js file that was created by browserify and now you will see that the file is much bigger and contains all the required dependencies. If we include this file in our index.html with the tag, start the application and open in the browser you will see that it works and you can send and receive messages. (See this step on github). We could even automate build steps (e.g. with gulp) but it’s out of scope for this post.

Summary

In this post, we looked at using the new SignalR JavaScript client in web applications. We learned how to use the client from both JavaScript and TypeScript. We tried to port an application using the previous version of SignalR to see how hard it is. In the next part, we will take a look at using the client in NodeJS applications.

Entity Framework 6 Easter of Love

While Entity Framework Core along with ASP.NET Core get all the hype today, Entity Framework 6 is still the workhorse of many applications running every day which won’t be converted to the Core world anytime soon, if at all. Because of this I decided to spend some time to give my EF extensions a small refresh to adapt to the changing landscape.

Github

Some of my extensions were hosted on Codeplex. I do most of the work on Github these days and Github is nowadays a de facto standard for open source projects. Codeplex not only looks dated but is also missing a lot of features Github has (searching the code on Github is far from perfect but Codeplex does not offer it at all). All in all this turned out to be the right decision given that it was recently announced that Codeplex is being shutdown. Anyways, here is where my projects previously hosted on Codeplex found their new homes:

Updating projects

I developed most of my EF extensions before Visual Studio 2015 was released. I found that opening them in Visual Studio 2015 was not a good experience – Visual Studio would update project/solution files automatically leaving unwanted changes. Therefore, I updated solution files to the version compatible with Visual Studio 2015. I also moved to a newer version of XUnit which does not require installing an XUnit runner extension in Visual Studio to enable running tests. Even though the solution files are marked as Visual Studio 2015 compatible they can be opened just fine with Visual Studio 2017 which shipped in the meantime.

New versions

This is probably the most exciting: I released new versions of a few of my extensions.

2nd Level Cache for Entity Framework

2nd Level Cache (a.k.a. EFCache) 1.1.0 contains only one new feature. This feature will, however, make everyone’s life easier. Until now the default caching policy cached results for all queries. In the vast majority of cases this behavior is not desired (or plainly incorrect) so you had to create your own policy to limit caching only to results from selected tables. In EFCache 1.1.0 you can specify store entity sets (i.e. which correspond to tables in the database) for which the results should be cached when creating the default caching policy. As a result you no longer have to create your own policy if you want to control simple caching. This change is not breaking.

Store Functions for Entity Framework

I received a couple of community Pull Requests which are worth sharing so yesterday I published on NuGet the new new version of the Store Functions for Entity Framework (1.1.0) containing these contributions. pogi-b added support for Built-in functions so you can now map built-in store functions (e.g. FORMAT or MAP) and use them in your queries. PaulVrugt added ability to discover function stubs marked as private. The first change is not breaking. If you happened to have private function stubs that were not discovered before (a.k.a. dead code) they will be discovered now as a result of the second change.

EF6 CodeFirst View Generation T4 Template for C#

Visual Studio 2017 now requires extensions to use VSIX v3 format. The EF6 CodeFirst View Generation T4 Template for C# extension used format v1 and could not be installed in Visual Studio 2017. I updated the VSIX format to v3 and dropped support for Visual Studio 2010 and 2012.

Note: I have not updated other view generation templates for EF4/EF5 to work with Visual Studio 2017. If you need them to work with VS 2017 let me know and I will update.

Happy Easter!

 

SignalR Core Part 2/3: ASP.Net Core Sockets

Disclaimer: SignalR Core is still in early stages and changing rapidly. All information in this post is subject to change.

To test some of the scenarios described in the first part of this mini-series I came up with an idea for a relatively simple application where users can report to the server the weather at their location and their report will be broadcast to all the connected clients. I called this application SocialWeather. The central part of the system is an ASP.Net Core application running SignalR Core server. The server can handle massages received in one of the following formats – JSON, Protobuf and pipe (the pipe format is a simple format I created where the data is separated by the pipe (|) and the message ends with the new line character (\n)). I also created 3 different clients – a JavaScript clients using the JSON format, a C# client using Protobuf and a lua client using the pipe format. The JavaScript client is part of a web page served by the same application that hosts the server. The C# client is a console application that that can send and receive Protobuf messages. The lua client is the most interesting as it runs on an ESP8266 development board with the NodeMCU firmware. The whole system is using the “socket” level of the new SignalR which is a kind of counterpart of persistent connection in the previous version of SignalR (so no hubs API here). All the clients use bare websockets to connect to the server (in other words there is no SignalR client involved).

The SocialWeather server, which includes the JavaScript client is a sample project in the SignalR repo and you can run it yourself – just clone the repo, run build.cmd/build.sh to install the correct version of the runtime and restore packages. Then go to the samples/SocialWeather folder and start the server with dotnet run.

The C# and lua clients are in my personal repo. Running the C# client is straightforward. After you clone the repo you need to restore packages update the URL to the SignalR server and run the client with dotnet run. Each time you press Enter the client will generate a random weather report and send it to the server. If you type “!q:” the client will exit.

The lua client is meant to run on an ESP8266 compatible board with NodeMCU firmware installed. Preparing the board to run the client requires a bit of work. The first step is to set up serial communication to the board. If the module you have is equipped with a USB port (I have the Lolin v3 board which does have a USB port) it should be enough to install a VCP (Virtual COM Port) drivers (you can find the drivers here http://www.ftdichip.com/FTDrivers.htm or here http://www.silabs.com/products/mcu/pages/usbtouartbridgevcpdrivers.aspx). If your board doesn’t have a USB port, you will need to use an additional module (e.g.  Arduino)  for USB-to-serial translation (you can find tutorials about setting it up on the web).

The next step is to make sure that your board has the right firmware. It needs to be a NodeMCU firmware with net, http, wifi, and websocket support. You can request a build here and follow steps in this tutorial to re-image your board. When the board is ready you should be able to connect to it using a serial terminal (I used screen on Mac and Putty on Windows). On Windows determine the COM port number the board is using using the Device Manager. On Mac it will be one of the /dev/tty* devices. The speed needs to be set to 115200. When using Putty make sure the Flow Control setting is set to None or the communication will not be working correctly.

puttyflowcontrol

Once on the device you need to connect it to your wireless network by sending the following commands (you need to replace SSID with the name of the network and PWD with the password or an empty string for open networks):

wifi.setmode(wifi.STATION)
wifi.sta.config(SSID, PWD)
wifi.sta.connect()

We are now ready to run the client. If you haven’t already, clone the repo containing the client go to src/lua-client folder and update the URL to the SocialWeather SignalR server. Now you can transfer the file to the device (if you are connected to the device with the terminal you need to disconnect). The nodemcu-uploader Python script does the job.

If all stars aligned correctly you should be able now to start the client by executing:

dofile(“social-weather.lua”)

it will print a confirmation message once it connected  successfully to the server and then will print weather reports sent by other client. You can also send your own weather report just by typing:

ws:send(“72|2|0|98052\n”)

and pressing Enter.

The command may look a bit cryptic but is quite simple. The ws is a handle to the websocket instance created by the social-weather.lua script. send is a method of the websocket class so we literally invoke the send method of websocket interactively. The argument is a SocialWeather report in the pipe format: a pipe separated list of values – temperature, weather, time, zipe code – terminated with \n.

This is what it looks like when you run all three clients:

social-weather

Let’s take a look at how things work under the hood. On the server side the central part is the SocialEndPoint class which handles the clients and processes their requests. If you look at the loop that processes requests it does not do any parsing on its own. Instead it offloads parsing to a formatter and deals with strongly typed instances. Formatter is a class that knows how to turn a message into an object of a given type and vice versa. In case of the SocialWeather application the only kind of messages sent to and from the server are weather reports so this is the only type formatters need to understand.

When sending messages to clients the process is reversed. The server gives the formatter a strongly typed object and leaves it to formatter to turn it into a valid wire format.

How does the server know which formatter to use for a given connection? All available formatters need to be registered in the DI container as well as mapped to a type they can handle and format. When establishing the connection, the client sends the format type it understands as a query string parameter. The server stores this value in the connection metadata and uses later to resolve the correct formatter for the connection.

On the client side things are even simpler. The lua client has just 30 lines of code half of which is concerned with printing weather reports in a human readable form. Because the format of the connection cannot change once the connection is established message parsing can be hardcoded. The rest is just setting up the websocket to connect to the server and react to incoming message notifications.

The C# client is equally simple. It contains two asynchronous loops – one for receiving messages (weather reports) from the server and one for sending messages to the server. Again, handling the wire format (which in this case is Protobuf) is hardcoded in the client.

This is pretty much all I have on ASP.Net Core Sockets. With a simple application we were able to validate that the new version of SignalR can handle many scenarios the old one couldn’t. We were able to connect to the server from different platforms/environments without using a dedicated SignalR client. The server was capable of handling clients that use different and custom message formats – including a binary format (Protobuf). Finally, all this could be achieved with a small amount of relatively simple code.

SignalR Core Part 1/3: Design Considerations

Disclaimer: SignalR Core is still in early stages and changing rapidly. All information in this post is subject to change.

A few months ago, we started working on the new version of SignalR that will be part of the ASP.Net Core framework. Originally we just wanted to port existing code and iterate on it. However, we soon realized that doing so would prevent us from enabling new scenarios we wanted to support. The original version of SignalR was designed around long polling (note that back in the day support for websockets was not as common as it is today – it was not supported by many web browsers, it was not supported in .NET Framework 4, it was not (and still isn’t) supported natively on Windows 7 and Windows 2008 R2). A JSON based protocol was baked in and could not be replaced which blocked a possibility of using other (e.g. binary) formats. Starting the connection was heavy and complicated – it required sending 3 HTTP requests whose responses had to be correlated with messages sent over the newly created transport (you can find a detailed description of the protocol in SignalR on the wire – an informal description of the SignalR protocol – a post I wrote on this very subject). This basically meant that a dedicated client was required to talk to a SignalR server. In the old design the server was centered around MessageBus – all messages and actions had to go through the message bus. This made the code very complex and error prone especially in scale-out scenarios where all the servers were required to have the same data. The state (e.g. cursors/message ids, groups tokens etc.) was kept on the client which would then send it back to the server when needed (e.g. when reconnecting). The need of keeping the state up-to-date significantly increased the size of the messages exchanged between the server and the client in most of the non-trivial scenarios.

In the new version of SignalR we wanted to remove some of the limitations of the old version. First, we decided to no longer use long polling as the model transport. Rather, we started with a premise that a full duplex channel is available. While this might sound a lot of like websockets we are thinking that it will be possible to take it further in the future and support other protocols like TCP/IP. Note, it does not mean that the long polling and server sent events transports are going away. Only, that we would not drag better transports down to the standards of worse transports (e.g. websockets supported binary format but long polling (until XmlHttpRequest2) and server sent events didn’t so in the old version of SignalR there was no support for binary messages. In the new version we’d rather base64 encode messages if needed and let users use what websockets offers). Second, we did not want to bake in any specific protocol or message format. Sure, for hub invocations we will still need to be able to get the name of the hub method and the arguments but we will no longer care how this is represented on the wire. This opens the way to using custom (including binary) formats. Third, establishing the connection should be lightweight and connection negotiation can be skipped for persistent duplex transports (like websockets). If a transport is not persistent or uses separate channels for sending and receiving data connection negotiation is required – it creates a connection id which will be used to identify all the requests sent by a given client. However, if there are no multiple requests because the transport is full duplex and persistent (like in the case of websockets) the connection id is not needed – once the connection is established in the first request it is used to transfer the data in both directions. In practice, this means that you can connect to a SignalR server without a SignalR client – just by using bare websockets.

There are also a few things that we decided not to support in the new SignalR. One of the biggest ones was the ability re-establish a connection automatically if the client loses a connection to the server. While it may not be obvious, the reconnect feature has a huge impact on the design, complexity and performance of SignalR. Looking at what happens during reconnect should make it clear. When a client loses a connection, it tries to re-establish it by sending the reconnect request to the server. The reconnect request contains the id of the last message the client received and the groups token containing the information about groups the client belongs to. This means that the server needs to send the message id with each message so the client can tell the server what was the last message it received. The more topics the client is subscribed to the bigger the message id gets up to the point where the message id is much bigger than the actual message.

Now, when the server receives a reconnect request it reads the message id and tries to resend all the messages the client missed. To be able to do that the server needs to keep track of all messages sent to each client and buffer at least some recent messages so that it can resend them when needed. Indeed, the server has a buffer per connection which it uses to store recent messages. The default size of that buffer is 1000 messages which creates a lot of memory pressure. The size of the buffer can be configured to make it smaller but this will increase the probability of losing messages when a reconnect happens.

The groups token has similar issues – the more groups the client belongs to the bigger the token gets. It needs to be sent to the client each time the client joins or leaves a group so the client can send it back in case of reconnects to re-establish group membership. The size of the token limits the number of groups a client can belong to – if the groups token gets too big the reconnect attempt will fail due to the URL being bigger than the limit.

While auto-reconnect will no longer be supported in SignalR users can build their own solution to this problem. Even today people try restarting their connection if it was closed by adding a handler to the Closed event in which they start a new connection. It can be done in a similar fashion in SignalR core. It’s true that the client will no longer receive messages it missed but this could happen even in the old SignalR – if the number of message the client missed was greater than the size of the message buffer the newest messages would overwrite the oldest messages (message buffer is a ring buffer) so the client would never receive the oldest messages.

Another scenario we decided not to support in the new version of SignalR was allowing clients to jump servers (a multi-server scenarios). Before, the client could connect to any server and then reconnect or send a data to any other server in the farm. This required that all servers had all the data to be able to handle requests from any client. The way it was implemented was that when a server receive a message it would publish it to all the other SignalR servers via MessageBus. This resulted in a huge number of messages being sent between SignalR servers.

(Side note. Interestingly, the scenario of reconnecting to a different server than the one the client was originally connected to often did not work correctly due to server misconfiguration. The connection token and groups token are encrypted by the server before sending them to the client. When the server receives the connection token and/or groups token it needs to be able decrypt it. If it cannot, it rejects the request with the 400 (Bad Request) error. The server uses the machine key to encrypt/decrypt the data so, all machines in the farm must have the same machine key or the connection token (which is included in each request) encrypted on one server can’t be decrypted on another server and the request fails. What I have seen several times was that servers in the farm had different machine keys so, reconnecting to a different server did not actually work.)

In the new SignalR the idea is that the client sticks only to one server. In multi-server scenarios there is a client to server map stored externally which tells which client is connected to which server. When a server needs to send a message to a client it no longer needs to send the message to all other servers because the client might be connected to one of them. Rather, it checks what server the client is connected to and sends the message only to this client thus the traffic among SignalR server is greatly reduced.

The last change I want to talk about, somewhat related to the previous topic, is removing the built-in scale-out support. In the previous version of SignalR there was one official way of scaling out SignalR servers – a scale out provider would subclass the ScaleoutMessageBus and leave all the heavy lifting to SignalR. It sounds good in theory but with the time it became apparent that with regards to scale-out there is no “one size fits all” solution. Scaling out an applications turned out to be very specific to the application goals and design. As a result, many users had to implement their own solution to scaling out their applications yet still paid the cost of the built-in scale-out (even when using just one server there is an in-memory message bus all messages go through). While, scale-out support is no longer built-in the project contains a Redis based scale-out solution that can be used as-is or as a guidance to create a custom scale-out solution.

These are I think the biggest design/architecture decision we have made. I believe that they will allow to make SignalR simpler, more reliable and performant.