Using Raspberry Pi to flash ESP8266

Necessity is mother of invention. Even though in this case there was not a lot of invention, there was necessity. I wanted to play with the ESP8266 module and I wanted it badly. Unfortunately, I could not find any FTDI in my drawers, so I had to find a different way of talking to my ESP8266. I quickly concluded that I could use either Arduino or Raspberry Pi. I decided to start with Arduino but for whatever reason could not make it work. A little desperate, I decided to try Raspberry PI. This, thanks to the information I found on the Internet, got me much further. I found a few articles scattered over the web and after combining various pieces of information I was able not only to talk to my ESP8266 but also reprogram it. I decided to write this post so that all the information I found is in one place. Let’s get started.

The idea is to connect an ESP8266 module to Raspberry Pi’s GPIO serial pins and then use a serial communication terminal like minicom or screen to talk to the module. Before we can do this, we need to reconfigure our Raspberry Pi so that the OS is not using the serial interface. The first step is to disable serial logins. The easiest way to achieve this is to use raspi-config. Start the the raspi-config with sudo raspi-config, go to Interfacing Options and then to Serial. You will be asked two questions – answer them as follows:

  • Would you like a login shell to be accessible over serial?
    Select: No
  • Would you like the serial port hardware to be enabled?
    Select: Yes

The second thing to do is to switch off sending bootup info to serial by removing references to ttyAMA0 in the /boot/cmdline.txt file – e.g.

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=PARTUUID=9815a293-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

becomes:

dwc_otg.lpm_enable=0 console=tty1 root=PARTUUID=9815a293-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Finally, you need to restart your Raspberry Pi for changes to take effect

sudo reboot now

Now, that we prepared our Raspberry Pi we need to wire the ESP8266 module. You need to power down Raspberry PI and connect your ESP8266 module as per the diagram below.

RPI_ESP8266_normal_mode


ESP8266 Raspberry Pi
GND       GND
3.3V      3.3V
RXD       UART_TXD
TXD       UART_RXD
CH_PD     3.3V (via a pull down resistor)

After wiring the ESP8266 module to the Raspberry Pi has been completed we can boot the Pi and connect to the ESP8266 using screen (I am using screen because it is installed by default but you can use minicom if you prefer):

screen /dev/ttyAMA0 115200

To see if we connected successfully let’s send some commands to the module (note: when using screen you need to press Enter and CTRL+J to send the command).

Check if AT system works (AT):

AT
OK

Display the version (AT+GMR)

AT+GMR
AT version:1.2.0.0(Jul  1 2016 20:04:45)
SDK version:1.5.4.1(39cb9a32)
Ai-Thinker Technology Co. Ltd.
Dec  2 2016 14:21:16
OK

Connect to WiFi (a series of commands)

AT+CWMODE_CUR=1
OK

AT+CWJAP_CUR="ssid","pwd"
WIFI CONNECTED
WIFI GOT IP
OK

AT+CWJAP_CUR?
+CWJAP_CUR:"ssid","18:a6:f7:23:9e:50",6,-52
OK

AT+CIFSR
+CIFSR:STAIP,"192.168.0.110"
+CIFSR:STAMAC,"5c:cf:7f:36:cd:31"

The full list of AT commands can be found on the espressif’s web site in the documents section – look for ESP8266 AT Instruction Set. Note that some commands from the document may not work if you are using an older version of the firmware.

We are able to communicate with our ESP8266. Now, let’s try updating the firmware. First, we need to power down the PI again and connect the GPIO0 pin of the ESP8266 to the PI’s ground pin. This will make the module run in the flash mode.

RPI_ESP8266_flash_mode

The next step is to install the esptool which we will use to transfer firmware files to the module:

sudo apt-get update
sudo apt-get install python
sudo apt-get install python-pip
sudo pip install esptool

Finally, we need a firmware that we will flash to the device. We can get a new firmware by cloning the espressif’s ESP8266_NONOS_SDK  repo:

git clone https://github.com/espressif/ESP8266_NONOS_SDK

After the repo has been cloned we need to go to the bin folder and run the following command:


esptool.py --port /dev/ttyAMA0 --baud 115200  write_flash --flash_freq 40m --flash_mode qio 0x0000 boot_v1.7.bin 0x1000 at/512+512/user1.1024.new.2.bin 0x7E000 blank.bin

The output of the command should look like this:

esptool.py v2.0.1
Connecting...
Detecting chip type... ESP8266
Chip is ESP8266
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 1MB
Flash params set to 0x0020
Compressed 4080 bytes to 2936...
Wrote 4080 bytes (2936 compressed) at 0x00000000 in 0.3 seconds (effective 121.5 kbit/s)...
Hash of data verified.
Compressed 427060 bytes to 305755...
Wrote 427060 bytes (305755 compressed) at 0x00001000 in 27.9 seconds (effective 122.3 kbit/s)...
Hash of data verified.
Compressed 4096 bytes to 26...
Wrote 4096 bytes (26 compressed) at 0x0007e000 in 0.0 seconds (effective 4538.5 kbit/s)...
Hash of data verified.

Once the process is complete we can power down the Raspberry Pi and disconnect the ESP8266 GPIO0 pin from GND to make it run again in the normal mode. Now, we can start the PI, and dump the ESP8266 firmware version with the AT+GMR command:


AT+GMR
AT version:1.4.0.0(May  5 2017 16:10:59)
SDK version:2.1.0(116b762)
compile time:May  5 2017 16:37:48
OK

This is a much newer version than what we originally had so we were able to flash our ESP8266 successfully. Yay!

Acknowledgements:

The following articles were instrumental in writing this post:

 

 

Advertisement

The SignalR for ASP.NET Core JavaScript Client, Part 2 – Outside the Browser

Last time we looked at using the ASP.NET Core SignalR TypeScript/JavaScript client in the browser. I mentioned, however, that the new client no longer has dependencies that prevent from using it outside the browser. So, today we will try taking the client outside the browser and use it in a NodeJS application. We will add a NodeJS client for the SignalR Chat service we created last time. Initially we will write the client in JavaScript and then we will convert it to TypeScript.

Let’s start from creating a new folder in the SignalRChat repo and adding a new node project:

mkdir SignalRChatNode
cd SignalRChatNode
npm init

We will call the application signarlchatnode and we will leave all other options set to default values. (6425ec1)

Our application will read messages typed by the user and send them to the server. To handle user input we will use node’s readline module. To see that things, work, let’s just add code to prompts the user for the name and displays it in the console. We will use it a starting point of our application (34bc493).

const readline = require('readline');
let rl = readline.createInterface(process.stdin, process.stdout)

rl.question('Enter your name: ', name => {
console.log(name);
  rl.close();
});

To communicate with the SignalR server we need to add the SignalR JavaScript client to the project using the following command (7875c07):

npm install @aspnet/signalr-client --save

We can now try starting the connection like this (3228a10):

const readline = require('readline');
const signalR = require('@aspnet/signalr-client');

let rl = readline.createInterface(process.stdin, process.stdout);

rl.question('Enter your name: ', name => {
  console.log(name);

  let connection = new signalR.HubConnection('http://localhost:5000/chat');
  connection.start()
  .catch(error => {
    console.error(error);
    rl.close();
  });
});

The code looks good but if you try running it, it will immediately fail with the following error:

Error: Failed to start the connection. ReferenceError: XMLHttpRequest is not defined
ReferenceError: XMLHttpRequest is not defined

What happened? The new JavaScript client no longer depends on the browser but still uses standard libraries like XmlHttpRequest or WebSocket to communicate with the server. If these libraries are not provided the client will fail. Fortunately, the required functionality can be easily polyfilled in the NodeJS environment. For now, we will just stick the polyfills on the global object. It’s not beautiful by any means but will do the trick. We are discussing how to make it better in the future but at the moment this is the way to go.

Depending on the features of SignalR you plan to use you will need to provide appropriate polyfills. Currently the absolute minimum is XmlHttpRequest. SignalR client uses it to send the initial OPTIONS HTTP request which initializes the connection on the server side and for the long polling transport. So, if use the long polling transport only, XmlHttpRequest is the only polyfill you will need to provide . If you want to use the WebSockets transport you will need a WebSocket polyfill in addition to XmlHttpRequest. (We are thinking about skipping sending the OPTIONS request for WebSockets. If this is implemented you will not need the XmlHttpRequest polyfill when using the WebSockets transport.) For ServerSentEvents transport you will need an EventSource polyfill. Finally, if you happen to use binary protocols (e.g. MessagePack) over the ServerSentEvent transport you will need polyfills for atob/btoa functions. For simplicity, we will use the WebSocket transport in our application so we will add only polyfills for XmlHttpRequest and WebSockets:

npm install websocket xmlhttprequest --save

and make them available globally via:

XMLHttpRequest = require('xmlhttprequest').XMLHttpRequest;
WebSocket = require('websocket').w3cwebsocket;

If we run the code now we will see something like this:

moozzyk:~/source/SignalRChat/SignalRChatNode$ node index.js
Enter your name: moozzyk
moozzyk
Information: WebSocket connected to ws://localhost:5000/chat?id=0d015ce4-3a78-4313-9343-cb6183a5e8ea
Information: Using HubProtocol 'json'.

which tells us that the client was able to connect successfully to the server. (946f85d)

Now, we need to add some code to handle user input and interact with the server and our Node SignalR Chat client is ready. (I admit that the user interface is not very robust but should be enough for the purpose of this post). You can now talk to browser clients from your node client and vice versa (0f7f71f):

Screen Shot 2017-09-30 at 6.57.14 PM

Now let’s convert our client to TypeScript. We will start from creating a new TypeScript project with tsc --init. In the generated tsconfig.json file we will change the target to es6. We will also add an empty index.ts file and delete the existing index.js file (we will no longer need the index.js file since we will now be generating one by compiling the newly created index.ts). (b83cf92) If you now run tsc you should see an empty index.js file created as a result of compiling the index.ts file.  The last thing to do is to actually convert our JavaScript code to TypeScript. We could just translate it one-to-one but we can do a little better. TypeScript supports async/await which makes writing asynchronous code much easier. Since many of SignalR client methods return Promises we can just await these calls instead of using .then/.catch functions. Here is how our node SignalRChat client written in TypeScript looks like (2a6d0e9):

import * as readline from "readline"
import * as signalR from "@aspnet/signalr-client"

(<any>global).XMLHttpRequest = require("xmlhttprequest").XMLHttpRequest;
(<any>global).WebSocket = require("websocket").w3cwebsocket;

let rl = readline.createInterface(process.stdin, process.stdout);

rl.question("Enter your name: ", async name => {
  console.log(name);
  let connection = new signalR.HubConnection("http://localhost:5000/chat");

  connection.on("broadcastMessage", (name, message) => {
    console.log(`${name}: ${message}`);
    rl.prompt(true);
  });

  try {
    await connection.start();
    rl.prompt();

    rl.on("line", async input => {
      if (input === "!q") {
        console.log("Stopping connection...");
        connection.stop();
        rl.close();
        return;
      }
      await connection.send("send", name, input);
    });
  }
  catch (error) {
    console.error(error);
    rl.close();
  }
});

You can run it by executing the following commands:
tsc
node index.js

Today we learned how to use the ASP.NET Core SignalR client in the NodeJS environment. We created a small node JavaScript application that was able to communicate with browser clients which. Finally, we converted the JavaScript code to TypeScript and learn a little bit about the TypeScript’s async/await feature.

The SignalR for ASP.NET Core JavaScript Client, Part 1 – Web Applications

The first official release of SignalR for ASP.NET Core – alpha1 – was just released. In this release, all SignalR components were rewritten to make SignalR simpler, easier to use and more reliable.

The SignalR JavaScript client has always been a fundamental part of SignalR. Unfortunately, it has a few limitations which made it hard to extend or use outside the browser. The rewrite allowed to introduce changes which allow to take the client outside the browser (no more dependency on jQuery, YAY!) and open new scenarios. And this is what this blog post will focus on. I split the post to two parts. In the first part I will show how to use the client in a web application from both JavaScript and TypeScript. In the second, part we will look at NodeJS.

The plan for this part is to recreate the chat application from the tutorial on the previous version of SignalR and then to convert it to use the new SignalR Server and JavaScript client. The sample is simple enough to allow us to focus on SignalR aspects rather than on application intricacies. As a bonus, we will see what the experience of porting an application from the previous version of SignalR is. I created a github repo for the application where each commit is a step described in this post. I will refer to particular commits from this post to show changes for a given step.

Setting up the Server

Let’s start from creating an empty ASP.NET Core application. We can do that from command line by running the dotnet new web command. (See this step on github).

Once the application is created we can start the server with dotnet run and make sure it works by navigating to http://localhost:5000 from a browser.

After we ensured that the application runs we can add SignalR server components. First, we need to add a reference to the SignalR package to the SignalRChat.csproj file (See this step on github).

Now we can add the Chat Hub class – we will just copy the code from tutorial and tweak a few things. This is how the hub class looks after the changes:

using System;
using Microsoft.AspNetCore.SignalR;
namespace SignalRChat
{
    public class ChatHub : Hub
    {
        public void Send(string name, string message)
        {
            // Call the broadcastMessage method to update clients.
            Clients.All.InvokeAsync("broadcastMessage", name, message);
        }
    }
}

The changes we made were only cosmetic – we removed the reference to the System.Web namespace, added 'Core' to the Microsoft.AspNet.SignalR so that it reads Microsoft.AspNetCore.SignalR. We also changed how we invoke the client-side method by passing the method name as the first parameter to the InvokeAsync call. (See this step on github).

Now that we created a hub we need to configure the application to be aware of SignalR and to forward SignalR related messages to our hub. It’s as easy as calling AddSignalR extension method in the ConfigureServices method of our Startup class and mapping the hub with the UseSignalR method. We will also add the static files middleware which will be responsible for serving static files. The Startup class should look like this:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddSignalR();
    } 

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseFileServer();

        app.UseSignalR(routes =>
        {
            routes.MapHub<ChatHub>("chat");
        });
    }
}

(See this step on github).

And this is all the work we had to do create a functional SignalR chat server. Now we can focus on the client side.

The JavaScript Client

In the new version of SignalR the JavaScript client is distributed using npm. The npm module contains a version of the client that can be just included in a web page using the tag, as well as, typings and modules that can be consumed from TypeScript. To get the client to your machine you need to install npm if you haven’t already and run:

npm install @aspnet/signalr-client

The client will be installed in the node_modules folder and you can find the necessary files to include in the node_modules/@aspnet/signalr-client/dist/browser folder. You may wonder why there are so many files in this folder and what purpose they serve. Let’s go over them then and explain.

First, you will find that there are two sets of files – files that contain ES5 in the names and files that do not contain ES5 in the names. SignalR JavaScript uses ES6 (a.k.a EcmaScript 2015) features like Promises or arrow functions. Not all browsers however, support ES6 (looking at you Internet Explorer). The files without ES5 in the names are meant to be used in browsers that support ES6. The files that contain ES5 in the names are the ES6 files transpiled to ES5. They are ES5 compatible and include all required dependencies. The downside of the ES5 files is that they are much bigger than ES6 files.

Another interesting set of files are files containing msgpackprotocol in the name. The new version of SignalR supports custom hub protocols – including binary protocols – and has built-in support for a binary protocol based on MessagePack. The JavaScript implementation of the MessagePack based hub protocol (using the msgpack5) turned out to be quite big so we moved it to a separate file. This way you can include the MessagePack hub protocol only if you want to use it and will not pay the price if you don’t care.

You will also find that each file has a min counterpart. These are just minified versions of the corresponding files. You will want to use the minified versions in production but debugging is much easier with non-minified files so you may want to use non-minified versions during development.

Finally, there is also the third-party-notices.txt file. These are notices for the msgpack5 library and its dependencies used in the MessagePack hub protocol implementation.

Using the SignalR JavaScript Client from JavaScript

Now, that we know a little bit about the JavaScript client let’s update our application to use it.

First, let’s copy all the files from the node_modules/@aspnet/signalr-client/dist/browser folder to a new ​scritps/signalr folder under the wwwroot. (See this step on github).

After the files are copied, let’s create the index.html file in the wwwroot folder and paste the contents of the html file from the tutorial. (See this step on github).

If you try to run the application at this point it will not work. The index.html has references to files like the jQuery library or the old SignalR client which don’t exist. Let’s fix that. Note that even though jQuery is no longer required to the new SignalR client I will continue to use it to minimize the number of changes I need to make. All in all this is not a tutorial on how to remove jQuery from your app so let’s not get sidetracked. Let’s start from sorting out the scripts situation. For jQuery, I will replace the link with the one to the jQuery CDN. For SignalR, I will replace the link to the signalR-2.2.1.min.js file with signalR-client-1.0.0-alpha1.js (feel free to use the ES5 version if you are using a browser that don’t support ES6 features) and remove the link to hubs since hub proxies are currently not supported. (See this step on github (github trick – notice that the link ends with ?w=1 – try removing it and see what happens. Very useful when reviewing some PRs)).

Now we can finally fix the code. Fortunately, this is not a lot of changes:

  • Instead of using proxies we will just create a new HubConnection
  • To register the callback for the client side broadcastMessage method we will use the on function
  • We will replace the done method used by jQuery deferreds to the then used by ES6 promises
  • We will invoke hub methods with the invoke function

(See this step on github).

That’s pretty much it. If you run the application now you should be able to send and receive messages.

Using the JavaScript Client from TypeScript

We now know how to use the new JavaScript SignalR client from JavaScript code. The SignalR client module contains also all necessary bits that make it possible to be consumed from TypeScript. To see how it works let’s take our chat application a bit further and convert it TypeScript.

First, make sure that you have a recent TypeScript compiler installed – run tsc --version from command line. If running the command fails or you have an older version installed install the latest one using this command:

npm install typescript -g

After installing or updating the typescript compiler we will initialize a new project by running

tsc --init

in the project folder. This will create a tsconfig.json file which will look like this:

{
  "compilerOptions": {
    "target": "es6",
    "module": "commonjs",
    "strict": true,
    "noImplicitAny": true
  }
}

after performing some cleanup. We will also add a new chat.ts file which we will leave empty for now. If you run the tsc command from project root you should see an almost empty chat.js file generated from your chat.ts file. (See this step on github).

Because we are using TypeScript and will bring dependencies using npm we will no longer need JavaScript files for the browser so let’s delete them. (See this step on github).

To be able to add and restore dependencies the client will need, let’s create a package.json file by executint the npm init command. We will leave default values for almost all settings except for the project name which needs to be lowercase.

PS C:\source\SignalRChat\SignalRChat> npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (SignalRChat) signalrchat
version: (1.0.0)
description:
entry point: (chat.js)
test command:
git repository:
keywords:
author:
license: (ISC)
About to write to C:\source\SignalRChat\SignalRChat\package.json:

{
"name": "signalrchat",
"version": "1.0.0",
"description": "",
"main": "chat.js",
"dependencies": {},
"devDependencies": {},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC"
}

Is this ok? (yes)
PS C:\source\SignalRChat\SignalRChat>

Now let’s add our dependencies – signalr-client, jquery and jquery typings (they enable using jquery from TypeScript). We will use the --save-dev option to save the dependencies as dev dependencies in the package.json file.

npm install @aspnet/signalr-client --save-dev
npm install jquery --save-dev
npm install @types/jquery --save-dev

We also need to install browserify – a tool which we will use to create the final script to be used by the browser:

npm install -g browserify

(See this step on github).

We can now start working on the code. First, we need to import the dependencies we are going to use. We can do that by adding the following two lines at the top of our chat.ts file:

import * as signalR from "@aspnet/signalr-client"
import * as $ from "jquery

Now we can move the script from our .html file to the .ts file. If you do that and play a little bit with the code you will notice that intellisense now tells you about class members and function parameters and if you press F12 (in Visual Studio Code) it will take you to the function header. Another thing, you will see is an error on line 5.  This TypeScript telling you that there is a type mismatch for the parameter passed to the jQuery val() function – the prompt() function can return null which is not a valid input for the val() function.

VSCodeSignalR

In our case we know that prompt will return string so we will just cast the result to string to suppress the error.

Since we moved the function to the .ts file we can now remove all the JavaScript code from our index.html file. We can also remove all the tags since we no longer depend on them to bring dependencies (we also already deleted the scripts). (See this step on github).

Let’s compile our chat.ts file now by running tsc command. If you look at the generated chat.js file you will notice that it looks pretty much the same as the source chat.ts file with some additional lines at the top. You will also notice that it does not have the required dependencies (i.e. signalr-client and jquery). This is where browserify comes into play. We will use browserify to generate the final version of the file with all the dependencies. Let’s run the following command (you may need to create the wwwroot/scripts folder if one does not exist) from the project folder:

browserify .\chat.js -o .\wwwroot\scripts\chat.js

Take a look at the chat.js file that was created by browserify and now you will see that the file is much bigger and contains all the required dependencies. If we include this file in our index.html with the tag, start the application and open in the browser you will see that it works and you can send and receive messages. (See this step on github). We could even automate build steps (e.g. with gulp) but it’s out of scope for this post.

Summary

In this post, we looked at using the new SignalR JavaScript client in web applications. We learned how to use the client from both JavaScript and TypeScript. We tried to port an application using the previous version of SignalR to see how hard it is. In the next part, we will take a look at using the client in NodeJS applications.

Entity Framework 6 Easter of Love

While Entity Framework Core along with ASP.NET Core get all the hype today, Entity Framework 6 is still the workhorse of many applications running every day which won’t be converted to the Core world anytime soon, if at all. Because of this I decided to spend some time to give my EF extensions a small refresh to adapt to the changing landscape.

Github

Some of my extensions were hosted on Codeplex. I do most of the work on Github these days and Github is nowadays a de facto standard for open source projects. Codeplex not only looks dated but is also missing a lot of features Github has (searching the code on Github is far from perfect but Codeplex does not offer it at all). All in all this turned out to be the right decision given that it was recently announced that Codeplex is being shutdown. Anyways, here is where my projects previously hosted on Codeplex found their new homes:

Updating projects

I developed most of my EF extensions before Visual Studio 2015 was released. I found that opening them in Visual Studio 2015 was not a good experience – Visual Studio would update project/solution files automatically leaving unwanted changes. Therefore, I updated solution files to the version compatible with Visual Studio 2015. I also moved to a newer version of XUnit which does not require installing an XUnit runner extension in Visual Studio to enable running tests. Even though the solution files are marked as Visual Studio 2015 compatible they can be opened just fine with Visual Studio 2017 which shipped in the meantime.

New versions

This is probably the most exciting: I released new versions of a few of my extensions.

2nd Level Cache for Entity Framework

2nd Level Cache (a.k.a. EFCache) 1.1.0 contains only one new feature. This feature will, however, make everyone’s life easier. Until now the default caching policy cached results for all queries. In the vast majority of cases this behavior is not desired (or plainly incorrect) so you had to create your own policy to limit caching only to results from selected tables. In EFCache 1.1.0 you can specify store entity sets (i.e. which correspond to tables in the database) for which the results should be cached when creating the default caching policy. As a result you no longer have to create your own policy if you want to control simple caching. This change is not breaking.

Store Functions for Entity Framework

I received a couple of community Pull Requests which are worth sharing so yesterday I published on NuGet the new new version of the Store Functions for Entity Framework (1.1.0) containing these contributions. pogi-b added support for Built-in functions so you can now map built-in store functions (e.g. FORMAT or MAP) and use them in your queries. PaulVrugt added ability to discover function stubs marked as private. The first change is not breaking. If you happened to have private function stubs that were not discovered before (a.k.a. dead code) they will be discovered now as a result of the second change.

EF6 CodeFirst View Generation T4 Template for C#

Visual Studio 2017 now requires extensions to use VSIX v3 format. The EF6 CodeFirst View Generation T4 Template for C# extension used format v1 and could not be installed in Visual Studio 2017. I updated the VSIX format to v3 and dropped support for Visual Studio 2010 and 2012.

Note: I have not updated other view generation templates for EF4/EF5 to work with Visual Studio 2017. If you need them to work with VS 2017 let me know and I will update.

Happy Easter!

 

SignalR Core Part 2/3: ASP.Net Core Sockets

Disclaimer: SignalR Core is still in early stages and changing rapidly. All information in this post is subject to change.

To test some of the scenarios described in the first part of this mini-series I came up with an idea for a relatively simple application where users can report to the server the weather at their location and their report will be broadcast to all the connected clients. I called this application SocialWeather. The central part of the system is an ASP.Net Core application running SignalR Core server. The server can handle massages received in one of the following formats – JSON, Protobuf and pipe (the pipe format is a simple format I created where the data is separated by the pipe (|) and the message ends with the new line character (\n)). I also created 3 different clients – a JavaScript clients using the JSON format, a C# client using Protobuf and a lua client using the pipe format. The JavaScript client is part of a web page served by the same application that hosts the server. The C# client is a console application that that can send and receive Protobuf messages. The lua client is the most interesting as it runs on an ESP8266 development board with the NodeMCU firmware. The whole system is using the “socket” level of the new SignalR which is a kind of counterpart of persistent connection in the previous version of SignalR (so no hubs API here). All the clients use bare websockets to connect to the server (in other words there is no SignalR client involved).

The SocialWeather server, which includes the JavaScript client is a sample project in the SignalR repo and you can run it yourself – just clone the repo, run build.cmd/build.sh to install the correct version of the runtime and restore packages. Then go to the samples/SocialWeather folder and start the server with dotnet run.

The C# and lua clients are in my personal repo. Running the C# client is straightforward. After you clone the repo you need to restore packages update the URL to the SignalR server and run the client with dotnet run. Each time you press Enter the client will generate a random weather report and send it to the server. If you type “!q:” the client will exit.

The lua client is meant to run on an ESP8266 compatible board with NodeMCU firmware installed. Preparing the board to run the client requires a bit of work. The first step is to set up serial communication to the board. If the module you have is equipped with a USB port (I have the Lolin v3 board which does have a USB port) it should be enough to install a VCP (Virtual COM Port) drivers (you can find the drivers here http://www.ftdichip.com/FTDrivers.htm or here http://www.silabs.com/products/mcu/pages/usbtouartbridgevcpdrivers.aspx). If your board doesn’t have a USB port, you will need to use an additional module (e.g.  Arduino)  for USB-to-serial translation (you can find tutorials about setting it up on the web).

The next step is to make sure that your board has the right firmware. It needs to be a NodeMCU firmware with net, http, wifi, and websocket support. You can request a build here and follow steps in this tutorial to re-image your board. When the board is ready you should be able to connect to it using a serial terminal (I used screen on Mac and Putty on Windows). On Windows determine the COM port number the board is using using the Device Manager. On Mac it will be one of the /dev/tty* devices. The speed needs to be set to 115200. When using Putty make sure the Flow Control setting is set to None or the communication will not be working correctly.

puttyflowcontrol

Once on the device you need to connect it to your wireless network by sending the following commands (you need to replace SSID with the name of the network and PWD with the password or an empty string for open networks):

wifi.setmode(wifi.STATION)
wifi.sta.config(SSID, PWD)
wifi.sta.connect()

We are now ready to run the client. If you haven’t already, clone the repo containing the client go to src/lua-client folder and update the URL to the SocialWeather SignalR server. Now you can transfer the file to the device (if you are connected to the device with the terminal you need to disconnect). The nodemcu-uploader Python script does the job.

If all stars aligned correctly you should be able now to start the client by executing:

dofile(“social-weather.lua”)

it will print a confirmation message once it connected  successfully to the server and then will print weather reports sent by other client. You can also send your own weather report just by typing:

ws:send(“72|2|0|98052\n”)

and pressing Enter.

The command may look a bit cryptic but is quite simple. The ws is a handle to the websocket instance created by the social-weather.lua script. send is a method of the websocket class so we literally invoke the send method of websocket interactively. The argument is a SocialWeather report in the pipe format: a pipe separated list of values – temperature, weather, time, zipe code – terminated with \n.

This is what it looks like when you run all three clients:

social-weather

Let’s take a look at how things work under the hood. On the server side the central part is the SocialEndPoint class which handles the clients and processes their requests. If you look at the loop that processes requests it does not do any parsing on its own. Instead it offloads parsing to a formatter and deals with strongly typed instances. Formatter is a class that knows how to turn a message into an object of a given type and vice versa. In case of the SocialWeather application the only kind of messages sent to and from the server are weather reports so this is the only type formatters need to understand.

When sending messages to clients the process is reversed. The server gives the formatter a strongly typed object and leaves it to formatter to turn it into a valid wire format.

How does the server know which formatter to use for a given connection? All available formatters need to be registered in the DI container as well as mapped to a type they can handle and format. When establishing the connection, the client sends the format type it understands as a query string parameter. The server stores this value in the connection metadata and uses later to resolve the correct formatter for the connection.

On the client side things are even simpler. The lua client has just 30 lines of code half of which is concerned with printing weather reports in a human readable form. Because the format of the connection cannot change once the connection is established message parsing can be hardcoded. The rest is just setting up the websocket to connect to the server and react to incoming message notifications.

The C# client is equally simple. It contains two asynchronous loops – one for receiving messages (weather reports) from the server and one for sending messages to the server. Again, handling the wire format (which in this case is Protobuf) is hardcoded in the client.

This is pretty much all I have on ASP.Net Core Sockets. With a simple application we were able to validate that the new version of SignalR can handle many scenarios the old one couldn’t. We were able to connect to the server from different platforms/environments without using a dedicated SignalR client. The server was capable of handling clients that use different and custom message formats – including a binary format (Protobuf). Finally, all this could be achieved with a small amount of relatively simple code.