Cloud enabled Commodore 64: Part I – Introduction

This is the first post in a series that talks about my recent project dubbed “Cloud enabled Commodore 64”. The project is an attempt to connect Commodore 64 to Cloud (Azure) and let it communicate with a variety of clients – both modern and vintage. Here is a demo of the final result:

I had the idea of connecting Commodore 64 to Cloud in my head for a really long time. When I initially tried to take a stab at it, it was apparent that I didn’t have skills necessary to build the hardware. When the C64 WiFi modem came out I have already moved on and did not pay attention. Until that one, dark autumn Friday night, when I was browsing ebay and came across a C64 WiFi modem which I promptly bought. To be honest I did not even spend too much time looking at the pictures and when it arrived, I was very surprised to notice that the modem is just a NodeMCU Esp8266 board that can be plugged in to the C64 User port. This was a very pleasant surprise because I was already familiar with NodeMCU – I used it for one of the first ASP.Net Core SignalR demos.

The main purpose of the original C64 WiFi modem and its firmware was to allow Commodore 64 owners access BBS’es around the world with software called Nova Term. This did does not appeal to me at all. Due to the times and place I grew up in I did not even have access to a landline and the only way to get new software was via swapping with other enthusiast either at school or over mail. I did spend a night playing with the modem and figured one can use this modem to connect to a BBS directly from a Mac or Linux using screen (or a PC using Putty). This convinced me that the modem works and that my idea suddenly became real.

How? – High level design

The modem pretty much dictated the design. The communication between C64 and the modem would use Serial protocol and the modem would require a firmware that would deal with all network related affairs.

Cloud enabled Commodore 64 design

I decided on a few principles upfront:

  • Modem firmware will only deal with network (i.e. will not include client specific logic)
  • C64 software will be written in 6502 assembly
  • C64 software will use built-in Serial routines (KERNAL)
  • I will use Azure as the cloud provider and will use SignalR

I decided to use 6502 assembly to remember the fun I had using it on my first computer when I was a kid. Right decision or not, it made me appreciate how much progress have been made with regards to programming since then.

The KERNAL routines for Serial protocol are infamous for their bugs but they work OK at low speeds. There are several routines that patch and fix KERNAL code responsible for serial communication and allow reliable transfers at 2400 or even 9600 bauds. After I started investigating, I quickly realized that researching this subject would be interesting but also very time consuming. While it would be nice to have faster serial communication, running at 1200 bauds felt to be fast enough for this project – let’s be honest, in today’s world 9600 bauds is as fast (or as slow) as 1200 bauds and I never thought this project could have any practical usages.

Finally, I decided to go with Azure simply because I am much more familiar with Azure than AWS or Google Cloud. I am also quite familiar with SignalR and how it allows building engaging demos with the canonical example of a chat app being something that might even seem a legit use case for a 40-year-old computer.

These were my thoughts before I embarked on this project. I will however revisit some of these decision in a future post where I will reflect on surprises I encountered and what I could have done differently.

Why?

As noted before – this project was never supposed to have any practical usages. I just thought it would be a fun hobby project. And it really was. Seeing it working live on a real hardware was a blast.

That’s it for today. Next time I will go over the environment I used to develop this.

Automatic Reconnection in the Swift SignalR Client

As of version 0.7.0 the Swift SignalR Client supports automatic reconnection. This means that, if configured, the client will try to re-establish the connection to the server if the connection was lost. This post explains how this feature works, how to enable it and what configuration options are available.

Automatic reconnects

There are many scenarios where restoring an interrupted connection automatically is important. For instance, mobile applications very often have to be able to deal with unstable network and it’s crucial for these apps to be resilient to network issues. Conceptually the solution to the problem is simple – when the connection is lost a new connection needs to be started. In case of the Swift SignalR Client, this could always be implemented by the code consuming the client (i.e. on the application side). It turns out however, that in practice, implementing this logic is quite hard. Given that this is a common request and is the implementation is tricky it made sense to add support for automatic reconnection to the client. An important thing to note however is that the client does not offer anything more than just restoring the connection. In other words, the client will make a few attempts to restart the connection if it was stopped due to an error but will not do anything more than that. If reconnecting succeeds the server will treat the connection as a completely new connection and will assign it a new connection id. The client will also not receive messages it might have missed when it was disconnected. Anyone who used the non-Core version of SignalR can notice that this a big change to how reconnection worked in the non-Core version where, upon reconnecting, the server would recognize that the client has reconnected and resend missed messages. This functionality can no longer be implemented in the client for the Core version of SignalR because it requires cooperation from the server side (e.g. the server needs to buffer messages) and that logic does not exist in the Core version of SignalR.

Automatic reconnection is disabled by default. The main reason for this is backward compatibility – existing applications did not expect the connection to try to reconnect automatically so, they could break if the feature was enabled by default. Automatic reconnection requires also a bit of additional work and handle new lifecycle events.

Basics

The easiest way to enable automatic reconnection is to use the new .withAutoReconnect() method available on the HubConnectionBuilder class.

When automatic reconnection is enabled the application may receive two additional events:

  • connectionWillReconnect – invoked when the connection was lost
  • connectionDidReconnect – invoked when the connection was successfully restored

By default, the client will make up to four attempts to restore the connection. The first attempt will be made immediately after the connection was lost. The next three attempts will take place respectively 2, 10 and 30 seconds after the previous unsuccessful attempt. If all four attempts are unsuccessful the client will give up, close the connection and invoke the connectionDidClose event.

When the connection is being restored the client will not allow to invoke any method that tries to send data to the server.

Connection is now restartable

Adding support for automatic reconnection made the connection restartable. Before, once the connection was stopped it was necessary to create a new instance to be able to connect to the server again. This is no longer the case – the same instance can be used to restart connection to the server after it was stopped. This could be especially useful when handling background/foreground transitions.

Advanced Scenarios

The default reconnect configuration can be customized. It is possible to change the number of attempts as well as the time intervals between the attempts. The easiest way to do this is to create a new instance of the DefaultReconnectPolicy class with an array of retry intervals and pass this policy to the .withAutoReconnect() method. The number of retry intervals in the array tells the client how many reconnect attempts it should make, while the interval values indicate the time to wait between the attempts.

If the default reconnect policy is not flexible enough it is possible to go even further and create a custom reconnect policy by creating a class that conforms to the ReconnectPolicy protocol. This protocol has just one method – nextAttemptInterval that takes a RetryContext and returns the time interval telling the client when the next reconnect attempt should happen. The RetryContext instance passed to the nextAttempInterval method contains information about the current reconnect – the time when the reconnect was initiated, the number of failed attempts so far and the original error that triggered the reconnect. To stop further reconnect attempts the nextAttempInterval method should return DispatchTimeInterval.never. To put the policy to work the policy needs to be passed to the .withAutoReconnect() method when configuring the connection.

Backward Compatibility (a.k.a. Kill Switch)

As noted above, writing reconnect logic turned out to be quite tricky. It also required modifying existing code that is executed even when automatic reconnects are disabled. This created a risk of introducing issues into existing scenarios. In case of running into a bug like this it is possible to go back to the previous behavior by using the .withLegacyHttpConnection() method on the HubConnectionBuilder when creating a new hub connection.

Conclusion

These are pretty much all the details needed to be able to use automatic reconnection in the Swift SignalR Client. My hope is that automatic reconnection will make lives of developers much easier.

Swift Client for the Asp.NET Core version of SignalR – Part 2: Beyond the Basics

In the previous post we looked at some basic usage of the Swift SignalR Client. This was enough to get started but far from enough for any real-world application. In this post we will look at features offered by the client that allow handling more advanced scenarios.

Lifecycle hooks

One very important detail we glossed over in the previous post was related to starting the connection. While starting the connection seems to be as simple as invoking:

hubConnection.start()
view raw Start.swift hosted with ❤ by GitHub

it is not really the case. If you run the playground sample in one go you will see a lot of errors similar to:

2019-07-29T16:05:00.987Z error: Attempting to send data before connection has been started.

What’s going on here? The start() method is a not blocking call and establishing a connection to the server requires sending some HTTP requests, so takes much more time than just running code locally. As a result, the playground code continues to run and try to invoke hub methods while the client is still working in the background on setting up the connection. Another problem is that there is actually no guarantee that the connection will be ever successfully started (e.g. the provided URL can be incorrect, the network can be down, the server might be not responding etc.) but the start() method never returns whether the operation completed succcessfully. The solution to these problems is the HubConnectionDelegate protocol. It contains a few callbacks that allow the code that consumes the client be notified about the connection lifecycle events. The HubConnectionDelegate protocol looks like this:
public protocol HubConnectionDelegate: class {
func connectionDidOpen(hubConnection: HubConnection)
func connectionDidFailToOpen(error: Error)
func connectionDidClose(error: Error?)
}

The names of the callbacks should make their purpose quite clear but let’s go over them briefly:

  • connectionDidOpen(hubConnection: HubConnection)
    raised when the connection was started successfully. Once this event happens it is safe to invoke hub methods. The hubConnection passed to the callback is the newly started connection
  • connectionDidFailToOpen(error: Error) – raised when the connection could not be started successfully. The error contains the reason of the failure
  • connectionDidClose(error: Error?) – raised when the connection was closed. If the connection was closed due to an error the error argument will contain the reason of the failure. If the connection was closed gracefully (due to calling the stop() method) the error will be nil. Once the connection is closed trying invoking a hub method will result in an error

To set up your code to be notified about hub connection lifecycle events you need to create a class that conforms to the HubConnectionDelegate protocol and use the HubConnectionBuilder.withHubConnectionDelegate() method to register it. One important detail is that the client uses a weak reference to the delegate to prevent retain cycles. This puts the burden of maintaining the reference to the delegate on the user. If the reference is not maintained correctly the delegate might be released prematurely resulting in missing event notifications.
The example chat application shows the usage of the lifecycle events. It blocks/unblocks the UI based on the events raised by hub connection to prevent the user from sending messages when there is no connection to the server. The HubConnectionDelegate derived instance is stored in a class variable to ensure that the delegate will not be released before the connection is stopped.

HubConnectionBuilder

The HubConnectionBuilder is a helper class that contains a number of methods for configuring the connection:

  • withLogging – allows configuring logging. By default no logging will be configured and no logs will be written. There are three overloads of the withLogging method. The simplest overload takes just the minimum log level which can be one of:
    • .debug (= 4)
    • .info (= 3)
    • .warning (= 2)
    • .error (= 1)

    When the client is configured with this overload all log entries at the configured or higher log level will be written using the print function. The user can create more advanced loggers (e.g. a file logger) by creating a class conforming to the Logger protocol and registering it with one of the other withLogging overloads

  • withHubConnectionDelegate – configures a delegate that allows receiving connection lifecycle events (described above)
  • withHttpConnectionOptions – allows setting lower level configuration options (described below)
  • withHubProtocol – used to set the hub protocol that the client will use to communicate with the server. Not very useful at the moment given that currently the only supported hub protocol is the Json hub protocol which is also used by default (i.e. no additional configuration is required to use this protocol)

HttpConnectionOptions

The HttpConnectionOptions class contains lower level configuration options set using the HubConnectionBuilder.withHubConnectionOptions method. It allows configuring the following options:

  • accessTokenProvider – used to set a token provider factory. Each time the client makes an HTTP request (currently – because the client supports only the webSocket transport – this happens when sending the negotiate request and when opening a webSocket) the client will invoke the provided token factory and set the Authorization HTTP header to:
    Bearer {token-returned-by-factory}
  • skipNegotiation – by default the first step the client takes to establish a connection with a SignalR server is sending a negotiate request to get the capabilities of the server (e.g. supported transports), the connection id which identifies the connection on the server side and a redirection URL in case of Azure SignalR Service. However, the webSocket transport does not need a connection id (the connection is persistent) and if the user knows that the server supports the webSocket transport the negotiate request can be skipped saving one HTTP request and thus making starting the connection faster. The default value is false. Note: when connecting to Azure SignalR service this setting must be set to false regardless of the transport used by the client
  • headers – a dictionary containing HTTP headers that should be included in each HTTP request sent by the client
  • httpClientFactory – a factory that allow providing an alternative implementation of the HttpClient protocol. Currently used only by tests

Azure SignalR Service

When working with Azure SignalR Service the only requirement is that the HttpConnectionOptions.skipNegotiation is set to false. This is the default setting so typically no special configuration is required to make this scenario work.

Miscellaneous

Limits on the number of arguments

The invoke/send methods have strongly typed overloads that take up to 8 arguments. This should be plenty but in rare cases when this is not enough it is possible to drop to lower level primitives and use functions that operate on arrays of items that conform to the Encodable protocols. These functions work for any number of arguments and can be used as follows:

hubConnection.invoke(method: "Add", arguments: [2, 3], resultType: Int.self) { result, error in
if let error = error {
print("error: \(error)")
} else {
print("Add result: \(result!)")
}
}

Variable number of arguments

The SignalR server does not enforce that the same client method is always invoked with the same number of arguments. On the client side this rare scenario cannot be handled with the strongly typed .on methods. In addition -similarly to the scenarios described above – there is a limit of 8 parameters that the strongly typed .on callbacks support. Both scenarios can be handled by dropping to the lower level primitive which uses an ArgumentExtractor class instead of separate arguments. Here is an example:

hubConnection.on(method: "AddMessage", callback: { argumentExtractor in
let user = try argumentExtractor.getArgument(type: String.self)
var message = ""
if argumentExtractor.hasMoreArgs() {
message = try argumentExtractor.getArgument(type: String.self)
}
print(">>> \(user): \(message)")
})

These are pretty much all the knobs and buttons that the Swift SignalR Client currently offers. Knowing them allows using the client in the most effective way.

Swift Client for the Asp.NET Core version of SignalR – Part 1: Getting Started

SignalR-Client-Swift is a SignalR client for the Core version of SignalR for applications written in Swift. It’s been around for a while and, although the work is still in progress, it is stable and usable enough to use it in real apps. The project is an open source project hosted on GitHub and has received number of contributions from the community (e.g. including big features like support for Swift Package Manager). Unfortunately, so far, the documentation for this client has been between scarce and non-existent making it harder to adopt it. This and the next post aim to fix this problem.

Before looking diving into code let’s talk about the current state of affairs. As I mentioned, the work is far from finished and some features you can find in other clients are not currently supported. The following is the list of major SignalR features that are currently not implemented:

  • Long Polling and Server Sent Events transports
  • non-Json based hub protocols (e.g. Message Pack)
  • restartable connections
  • KeepAlive messages

Here is the more positive list of major features that are implemented:

  • webSockets transport
  • client and server hub method invocations (using Json hub protocol)
  • streaming methods
  • support for Azure SignalR Service
  • authentication with auth tokens

SignalR for ASP.Net Core is not backwards compatible with the previous version of SignalR. Hence, the Swift SignalR client will not work with the non-Core version of SignalR server.

Installation

The first step to use the client in a project is installation. Currently there are three ways to install Swift SignalR Client into your project:

CocoaPods

Add the following lines to your Podfile:

use_frameworks!
pod 'SwiftSignalRClient'
view raw CocoaPods hosted with ❤ by GitHub

Then run:
pod install

Swift Package Manager (SPM)

Add the following to your Package dependencies:

.package(url: "https://github.com/moozzyk/SignalR-Client-Swift", .upToNextMinor(from: "0.6.0")),

Then include "SignalRClient" in your target dependencies. For example:

.target(name: "MySwiftPackage", dependencies: ["SignalRClient"]),

Manually

Pull the code from the GitHub repo and configure SignalR client as an Embedded Framework.

Usage

Once the client has been successfully installed it is ready to use. The usage of the Swift SignalR Client does not differ much from other existing clients – you need to create a hub connection instance that you will use to connect and talk to the server. Note that you need to use the same instance of the client for the entire lifetime of your connection.
The easiest way to create a HubConnection instance is to use the HubConnectionBuilder class which contains a number of methods that allow configuring the connection to be created. For instance, creating a HubConnection instance with logging configured at the debug level would look like this:

let hubConnection = HubConnectionBuilder(url: URL(string: "http://localhost:5000/playground")!)
.withLogging(minLogLevel: .debug)
.build()

Creating a hub connection does not automatically start the connection. It just creates an instance that will be used to communicate with the server once the connection is started. This pattern makes it possible to register handlers for the client-side methods without risking missing invocations received between starting the connection and registering the handler. Handlers for the client-side methods are registered with the on method as follows:

hubConnection.on(method: "AddMessage") {(user: String, message: String) in
print(">>> \(user): \(message)")
}
view raw On.swift hosted with ❤ by GitHub

It is worth noting that types for the handler parameters must be specified and must be compatible with the types of values sent by the server (e.g. if the server invokes the method with a string the parameter type of the handler cannot be Int). The number of handler parameters should match the number of arguments used to invoke the client-side method from the server side.

After registering handlers it’s time to start the connection. It is as easy* as:

hubConnection.start()
view raw Start.swift hosted with ❤ by GitHub

From this point on, if the connection was started successfully, the handlers for the client-side methods will be invoked whenever the method was invoked on the server. Starting the connection allows also to invoke hub methods on the server side. (Trying to invoke a hub method on a non-started connection results in an error). SignalR supports two kinds of hub methods – regular and streaming. When invoking a regular hub method, the client may choose to be notified when the invocation has completed and receive the result of invocation (if the hub method returned any) or an error in case of an exception. Below are examples of such invocations:

// invoking a hub method and receiving a result
hubConnection.invoke(method: "Add", 2, 3, resultType: Int.self) { result, error in
if let error = error {
print("error: \(error)")
} else {
print("Add result: \(result!)")
}
}
// invoking a hub method that does not return a result
hubConnection.invoke(method: "Broadcast", "Playground user", "Sending a message") { error in
if let error = error {
print("error: \(error)")
} else {
print("Broadcast invocation completed without errors")
}
}
view raw Invoke.swift hosted with ❤ by GitHub

When invoking a hub method that returns a result providing the type of the result is mandatory and this type has to be compatible with the type of the value returned by the hub method. Also, there is no distinction between local and remote handlers – i.e. the completion handler will be called with an error not only when the method on the server side fails but also when initiating the invocation fails (e.g. when trying to invoke a method when the connection is not running).

Hub methods can also be invoked in a fire-and-forget manner. When invoking a hub method in this fashion the client will not be notified when the invocation has completed and will not receive any further events related to this method – be it a result or an error. The code below shows how to invoke a hub method in a fire-and-forget manner:

hubConnection.send(method: "Broadcast", "Playground user", "Testing send") { error in
if let error = error {
print("Send failed: \(error)")
}
}
view raw Send.swift hosted with ❤ by GitHub

Note, that the send method still takes a callback that allows handling errors but this callback will be called only for local errors – i.e. errors that occurred when sending data to the server.

SignalR streaming hub methods return a (possibly infinite) stream of items. Each time the client receives a new stream item a user provided callback will be invoked with the received value.
When a streaming method completes executing a completion callback will be invoked (except for this bug which I found writing this post). The client method that invokes streaming hub methods returns a stream handle. This handle can be used to cancel the streaming hub method. The following code snippet illustrates how to invoke and cancel a streaming hub method:

let streamHandle = hubConnection.stream(method: "StreamNumbers", 1, 10000, itemType: Int.self,
streamItemReceived: { item in print(">>> \(item!)") }) { error in
print("Stream closed.")
if let error = error {
print("Error: \(error)")
}
}
DispatchQueue.main.asyncAfter(deadline: .now() + .seconds(2)) {
hubConnection.cancelStreamInvocation(streamHandle: streamHandle) { error in
print("Canceling stream invocation failed: \(error)")
}
}
view raw Stream.swift hosted with ❤ by GitHub

If you no longer want to receive notifications from the server or invoke hub methods you can disconnect from the server with:

hubConnection.stop()
view raw Stop.swift hosted with ❤ by GitHub

One final note about types of arguments and results. The types of all the values sent to the server must conform to the Encodable protocol. The types for the values returned from the server must conform to the Decodable protocol. The most common types in Swift already conform to the Codable protocol (which means that they conform to both the Encodable and the Decodable protocols) and when creating custom structs/classes it is easy to make them conform to the Codable protocol as long as all the member variables already conform to the Codable protocol.

These are the basics of the SignalR Swift Client. The project repo contains additional resources in form of example applications for macOS and iOS. I also created a Swift playground which contains all code snippets published in this post. In the next post we will look at the connection lifecycle events, available configuration options and more advanced scenarios.

* – it is actually not entirely true but we will return to it in the second post†

Hello, OCaml!

I am a huge fan of the Advent of Code. I eagerly wait for it every year and once it starts I try to solve the problems the same day they appear. The nice thing about the problems is that they are initially relatively simple and get more difficult over time (and if you ever get stuck there is a subreddit for each problem that you can use to unblock yourself). This makes it a fantastic opportunity to try new languages. (Note, I intentionally said “try” and not “learn” because the vast majority of the problems can be solved in less than 100 lines of code and almost never require advanced data structures and/or language features). And this is what I do – each year I pick a language I have never used before and solve all the problems using this language. Two years ago – the first edition of the Advent of Code – I tried Scala, last year I tried Go and this year I ended up picking OCaml (with Rust and D being other contenders). One thing I decided to do differently this year however, was to write down my observations and things I struggled with to share them, hoping that it will help people trying to learn OCaml.

Preparing Development Environment

Before I could start solving the problems I needed to setup my dev environment and figure out how to build and run programs written in OCaml. I used Visual Studio Code as my editor because I was sure it would have an extension for OCaml (I used OCaml & Reason IDE). To build my programs I ended up using ocamlbuild although probably it was an overkill – I never had more than one file to compile and used at most one additional library (Str). With my dev environment set up I was ready to start solving Advent of Code problems and learning OCaml the hard (i.e. trial-and-error) way.

Common errors

When you try to pick a new language, you will initially make a lot of simple, syntactic mistakes. I think this was one of the biggest barriers to me at the beginning. I would try to compile my program and the compilation would fail with an error I could only stare at. Understanding the error would initially take me a lot of time and experimenting (like commenting out code etc.). After a few days I figured what errors my most common mistakes resulted in and things got much easier, but I would still occasionally encounter an error which took a relatively long time to resolve. (Note, some of the mistakes could probably have been avoided if I had read more on the language before I started coding but this is not the way I learn – I prefer reading just enough to be able to do simple things and then figure out things as I go).
Before I dive deeper into errors I encountered I would like to give a few hints that can help in locating and understanding the error:

  • The location of the error contains the line and the column where the error occurs. The column can be very helpful – especially when you invoke a function with multiple parameters or inline a function invocation
  • Oftentimes the mistake is not in the line the error message points to. If you can’t find anything wrong with the line the error message points to check the line(s) the function is invoked from
  • Learn how to read function signatures (e.g. int -> int -> int) to easier understand errors caused by passing values of incorrect types. See the Types of Functions in the OCaml tutorial.

During my adventure with OCaml I compiled a list of errors I encountered.

Syntax error

This was initially the most common error I saw. It can have many causes and I am sure that the list below is not exhaustive. Most common causes:

  • missing in after variable declaration:
    let sum a b =
    let s = a + b
    print_int
    (* Error: Syntax error *)
    let sum a b =
    let s = a + b in
    print_int s
    (* val sum : int -> int -> unit = <fun> *)
    view raw missing_in.ml hosted with ❤ by GitHub
  • using => instead of -> (likely specific to developers who mostly use C# or JavaScript/TypeScript)
  • using reserved words as variable names (I fell a few times for val and match I wanted to use respectively for a variable storing a temporary value and a result of regular expression match)
  • missing -> in pattern matching (the exact error is Syntax error: pattern expected

Unbound value

This error means that the function you are trying to call cannot be found. Most common causes:

  • You have a typo in the function name
  • Your recursive function is not marked rec:
    let factorial n =
    if n = 0 then 1
    else n * factorial(n - 1)
    (* Error: Unbound value factorial *)
    let rec factorial n =
    if n = 0 then 1
    else n * factorial(n - 1)
    (* val factorial : int -> int = <fun> *)
    view raw missing_rec.ml hosted with ❤ by GitHub

This function has type X It is applied to too many arguments; maybe you forgot a `;'.

Similarly to Syntax error there can be multiple causes for this error:

  • You actually forgot a ; to separate your statements:
    print_string "a"
    print_string "b"
    (* Error: This function has type string -> unit
    It is applied to too many arguments; maybe you forgot a `;'. *)
    print_string "a";
    print_string "b"
    (* Result: ab *)
  • You passed too many arguments to a function:
    let sum list = List.fold_left (fun v a -> a + v) 0 list
    sum [1;2;3] "a"
    (* Error: This function has type int list -> int
    It is applied to too many arguments; maybe you forgot a `;'. *)
    sum [1;2;3]
    (* Result: 6 *)
  • You forgot parenthesis when inlining a function (similar to a previous case but sometimes harder to notice):
    print_int max 4 5
    (* Error: This function has type int -> unit
    It is applied to too many arguments; maybe you forgot a `;'. *)
    print_int (max 4 5)
    (* Result: 5 *)

WTF errors

The following errors fall into one of the categories above but can be very hard to spot a beginner (like I was):

  • Arithmetical operations, string or list concatenation must be in parentheses when used as an argument to a function :
    print_int 2 + 3
    (* Error: This expression has type unit but an expression was expected of type
    int *)
    print_int (2 + 3)
    (* Result: 5 *)
    print_string "a"^"b"
    (* Error: This expression has type unit but an expression was expected of type
    string *)
    print_string ("a"^"b")
    (* Result: ab *)
  • Negative values also should use parentheses:
    print_int -5
    (* Error: This expression has type int -> unit
    but an expression was expected of type int *)
    print_int (-5)
    (* Result: -5 *)
  • Sometimes the error points to a line that does not seem to be the cause of the problem. For instance in the following example I did not convert the argument of the sqrt function to the float type but the error points to a totally different line:
    let is_prime n =
    let rec is_prime_aux n div =
    if div > int_of_float (sqrt n) then true
    else if n mod div = 0 then false
    else is_prime_aux n (div + 1)
    in
    is_prime_aux n 2
    (* Error: This expression has type float but an expression was expected of type
    int
    Points to `n` in `if n mod div...` *)
    let is_prime n =
    let rec is_prime_aux n div =
    if div > int_of_float(sqrt (float_of_int n)) then true
    else if n mod div = 0 then false
    else is_prime_aux n (div + 1)
    in
    is_prime_aux n 2
  • Errors caused by partial application triggered by unintentionally passing fewer parameters that the function requires.

Observations

If you don’t want a biased opinion about OCaml you can stop reading now 🙂

Overall, I did not enjoy OCaml as a programming language. Initially, I struggled with errors that did not have much meaning to me. Once I got past this phase I found that I was not very productive – even conceptually simple problems required too much code for my taste. Maybe part of it was me learning the language but there were other people doing Advent of Code in OCaml and I found their solutions rarely required less code than mine. I also thought that it might be because I don’t use functional programming in my day-to-day work (except for quasi-functional features of C# like LINQ) and can’t switch to the functional paradigm but I briefly looked at my solutions in Scala from 2015 and they are generally much more compact.

Because I wanted to learn the language and not the libraries I wanted to get away as much as possible with just the basic language features (i.e. without using external libraries). It turned out to be hard. One of the most basic operation is writing a formatted string. OCaml offers a number of print_* functions. Unfortunately, these functions are very basic and even writing a number followed by a new line requires two print statements. Printing a formatted string with these functions was so cumbersome that I eventually decided to use the Printf module.

Another thing that baffled me at the very beginning was reading file contents. Most Advent of Code problems require reading the puzzle input from a file and I could not find any simple method to read lines from a file. It took me more time than I wanted to spend on this basic problem. All solutions seemed overly complicated. I eventually found this function on stackoverflow:

let read_lines name : string list =
let ic = open_in name in
let try_read () =
try Some (input_line ic) with End_of_file -> None in
let rec loop acc = match try_read () with
| Some s -> loop (s :: acc)
| None -> close_in ic; List.rev acc in
loop []

I have to admit that being a novice I initially did not fully understand how it worked but hey, it did work. The other alternatives I found were either even more complicated or required understanding concepts I did not event want to learn at the time (e.g. channels). All in all, I have to say there is a heck of complexity to achieve a very basic task that in many languages is just a single, self-explanatory line of code – e.g. File.ReadAllLines("myFile.txt").

I was also annoyed but by the lack of functions for simple string processing. To treat a string to as a sequence of characters you need to do something like this (again, I am not the author of this function):

let explode s =
let rec exp i l =
if i < 0 then l else exp (i - 1) (s.[i] :: l) in
exp (String.length s - 1) []

In reality, it’s hard to do string processing without using the Str module (btw. compare the name with the String module that contains basic string operations and is part of the standard library and tell me how you would not be confused which one is which) which supports regular expressions.

Speaking of regular expressions I found them weird to use. The first weird thing was that you could not just get all matched groups (e.g. as a list). Rather, you need to enumerate them one by one without knowing how many groups where matched (so you potentially enumerate until you hit an error). The other weird thing was that it seems that matched groups (or maybe some state that allows calculating them) are stored in the library and are keyed by the original string that was processed. Each time you want to get a matched group you need to provide this original string.

API inconsistencies are also irritating. For instance, both List and Hashtbl have the fold_left function. However, while List takes the function, the initial value and the list as parameters, Hashtbl takes the function, the hash table and the initial value (i.e. the initial value and the container are swapped comparing to List). It seems like a small thing, but I hit this many, many times.

Similarly, if you have multiple statements you need to separate them with a semicolon (;). However, if you use multiple statements in the then or else clause they also need to be wrapped with begin/end (or parentheses). On the other hand, you don’t need begin/end if you have multiple statements inside a loop (yes, I know, I should use recursion but sometimes using a loop is, you know, just simpler).

Runtime errors are a nightmare. You don’t get any details about the error except for the message. This might work if an exception is thrown from your own code but how you are supposed to find the bug effectively if all you get is:
Fatal error: exception Invalid_argument("index out of bounds")?

The last thing that I was surprised by was how scarce the documentation for OCaml is. When I decided to use OCaml to solve Advent of Code problems I knew it was not one of the mainstream programming languages, but I did not consider it completely niche. As soon as I started I realized that there is only a very limited number of resources at my disposal. Ironically, one of the best turned out to be the library reference on the https://caml.inria.fr website whose main page says:
This site is updated infrequently. For up-to-date information, please visit the new OCaml website at ocaml.org.
The library reference looks a bit raw and dated (e.g. see: https://caml.inria.fr/pub/docs/manual-ocaml/libref/String.html) and at the beginning I had a hard time digesting the information it provided but once I got more familiar with OCaml I would visit it all the time (yes, 50% of my visits were to check if the initial value for the fold_left function should go before or after the container).

To sum up – the next time I start a new project and have freedom to choose the language for the project I don’t think OCaml will be on the list. I am glad I tried it but I hoped for a more pleasant ride.

Coincidentally, HackerRank released their 2018 Developer Skill Report recently and OCaml was one of only two languages with negative sentiment among developers of all age groups. I will just say that the other language was Perl.

P.S. My solutions to Advent of Code 2017 can be found on github.