Monthly Archives: August 2012

Yes, you can! (create mutifile zip files in .NET Framework (finally))

I recently have been working on moving some projects from an old legacy build system to a new shiny build system. After understanding intricacies of the old build system I was able to isolate and remove 99% stuff we did not really need and comment out the remaining 1% we needed but could not make it work easily. I felt proud of myself because I was even able to build something in the new build system (OK, maybe the stuff we did not need was actually 98.9%?). So, I started uncommenting things we needed but were not as straightforward to fix. One of the first things I came across was the need of zipping files after building. I thought I would just use the MSBuild Zip task and call it done. Unfortunately it turned out that out of the box MSBuild does not have a Zip task. I could find it in MSBuild community tasks but for certain reasons we could not take a dependency on these. We also did not have a zip-like command line tool we could take a dependency on. One of the last sunny days around Seattle this year started looking gloomy to me. I asked folks on our team and they had a few ideas like using PowerShell (we don’t really need it anywhere else so again I wanted to avoid it), leveraging a third party library we could use to create a task or finally using GZipStream. I liked the idea of using GZipStream – I could create and inline MSBuild task which would be clean and simple. However gzip only supports compressing one file and I needed to zip several files so, I could not use it. I spent more time looking for something suitable and thought I found it – ZipPackage!. Again, my hopes were quickly dashed. This guy needs a metadata file containing content types of the included files. This file is then added to the archive. But I needed just a POZF™ (Plain Old Zip File) without any additional crap inside! Afternoon was getting really dark. I started looking at the option of using a third party library. It would add complexity to the build system. I would have to build tools before building projects. Suddenly, I had this feeling that instead of having a lightweight, easy and fast build we will have a new version of what we had before. Each time I wanted to start working on this my hands refused and decided to open a browser and check the news or something along these lines. So, I gave one more shot at finding a better solution but I could only find that there is an MSBuild Zip task in the community tasks and that the GZipStream basically can compress only one file and that the ZipPackage requires some metadata. I was loosing hope when suddenly on Xth page of results (where X > 5) I found this – the ZipArchive class. The ZipArchive is a new class in the .NET Framework 4.5 that allows creating zip files containing not only multiple files but also the whole folder hierarchy! In just one second all the clouds were gone. If there had not been for pierogi I would have probably stayed playing with this class until my MSBuild problem was resolved. On the other hand I don’t like when pierogi are too hot so I stayed a few more minutes just to see if it was easy to create a zip file. Here is what I came up with (I checked only briefly but could not find an example how to create a new archive on msdn):

public static void Main()
{
    Zip(
        new string[] 
        { 
            @"C:\Temp\file1.txt", 
            @"C:\temp\file2.txt", 
            @"C:\Source\ConsoleApplication1\ConsoleApplication1\Program.cs" 
        },
        @"c:\temp\out.zip");
}

static void Zip(string[] inputFileNames, string outputFileName)
{
    using (var outputFileStream = new FileStream(outputFileName, FileMode.Create))
    {
        using (var archive = new ZipArchive(outputFileStream, ZipArchiveMode.Create))
        {
            foreach (var inputFile in inputFileNames)
            {
                AddFileToArchive(archive, inputFile);
            }
        }
    }
}

static void AddFileToArchive(ZipArchive archive, string inputFile)
{
    const int BufferSize = 64 * 1024;

    // relies on inputFile being an absolute path 
    // - Substring(3) removes :\ from the path
    var archiveEntry = archive.CreateEntry(inputFile.Substring(3));

    using (var fs = new FileStream(inputFile, FileMode.Open))
    {
        using (var zipStream = archiveEntry.Open())
        {
            byte[] buffer = new byte[BufferSize];
            int bytesRead = -1;
            while ((bytesRead = fs.Read(buffer, 0, BufferSize)) > 0)
            {
                zipStream.Write(buffer, 0, bytesRead);
            }
        }
    }
}

I can open the archive I created in the Windows Explorer, browse and open files. No additional metadata files are added to the archive – it’s a POZF™. So – yes, you now can create a multifile zip archives in .NET Framework without using third party libraries. And now creating an inline MSBuild Zip task that fits my new build system is easy. This makes me happy!

Pawel Kadluczka

Advertisements

Digital Clock on Arduino (Uno) with Nokia LCD Display

When I heard about Arduino a couple years ago (yes, I was a bit late to the party) I immediately fell in love with it. The main reason was that it allowed me to try things that otherwise would require a lot of time and effort from a total electronics noob like me. With Arduino, a breadboard, a few LEDs and a couple of cables I was ready to write my very first program – controlling the LED with Arduino. It was fun! But I wanted to do something more interesting. I figured that while flashing a LED is fun displaying something on a screen must be even more fun. Soon I found that it was possible to buy a cheap Nokia 5110 LCD displays on ebay. I had some doubts whether I would be able to make it work but figured out that even if I wouldn’t I could always use it as a key chain or a similar geeky gadget. Indeed, when the display arrived I did have some problems to make it work. First, I was not sure how to connect the display to Arduino. Even though I found a few websites that showed how to do that I had hard time to make it work. I also tried a couple of sample programs that in theory should display something on the display but in practice they did not. Eventually, I found that one of the cables were connected incorrectly and the sequence to initialize the display did not work (at least on my display). Seeing something on the screen was a huge step forward and I finally could think about using the display for my own purpose. I decided that I could do a digital clock as it seemed a reasonably sized project which would still require understanding how the display really works. First I wanted to “design” fonts for the clock. For this I just used graph paper and this reminded me designing sprites for my C-64 in the late eighties. After designing all digits I had to understand how to encode these so that they could be drawn easily on the screen. I found the datasheet for the PCD8544 which is a controller for the display. This was the first time in my life I had to read a datasheet for an electronic component but I was able to understand what the most important things and the datasheet turned out to be an interesting read. Soon, I found what I wanted – the display has 6 rows and 84 columns. Each column in a row has 8 pixels. To draw something on a screen first you need to set a pointer (row and column) and then write a byte (8 bits) to draw the column. When you draw a column the pointer automatically advances to the next column – this is handy since you don’t really have to set the pointer all the time if you draw something like a bitmap. The mechanics of drawing on the screen surprised me. Firstly, I found it kind of similar to advanced graphics modes on 8-bit computers. Secondly, it was interesting to see that the programmer’s convenience totally lost with the low level design of the controller (or may be I am just spoiled by higher level APIs which allow me just to pass x and y to draw a point). After figuring out how drawing looks like encoding my digits was relatively simple. Now that I was able to draw digits on the screen it was the time to implement the clock. Initially I wanted to use the loop function but I quickly realized that it would not really work for a clock. I did not know how long drawing would take and therefore the clock would not work correctly. I could calculate how much time drawing took and then cut the wait time accordingly but it didn’t feel like the right solution. Obviously, there was one more (and I believe the only correct for this kind of problem) solution – interrupts. The only problem was to set everything up correctly. I spent some time reading Arduino datasheet but things did not really want to work. Fortunately I found this page which helped tremendously. After setting up interrupts things went smoothly and soon I had my digital clock up and running. I actually ended up using both the loop function and the interrupt. The interrupt just increases the timer while all the drawing takes place in the loop function. This way I don’t have to worry that drawing takes too much time and when a new interrupt happens the previous one is still being handled. That’s pretty much it. Here is a short video showing the clock in action:

If you want to try my project here is how I connected the Nokia 5110 display to Arduino:

Arduino Uno               Nokia 5110 Display
3.3V   ------------------ 1-VCC
PIN #7 ------------------ 3-SCE
PIN #6 ------------------ 4-RST
PIN #5 ------------------ 5-D/C
PIN #4 ------------------ 6-DNK(MOSI) (SDIN)
PIN #3 ------------------ 7-SCLK

I also posted the sketch on my github.

One more thing I always wanted to do but never really have had time to do was to add some buttons to enable setting the clock – at the moment the clock always starts at 00:00. Maybe one day I will come back to this project again (like I did today just to publish it) and will add the buttons…

Pawel Kadluczka

Unit testing XSD schemas

Once in a while a new task no-one is really eager to work on pops-up. From my experience in teams that are not focusing on or use extensively Xml related technologies most (if not all) tasks that have anything to do with XSD schemas belong to this group. This was the case in our team recently and I ended up to be a “volunteer” since the schedule was tight and I previously worked on the managed Xml team. So, I started refreshing my rusted XSD skills and soon I got something that more or less worked. It was a good starting point but then I asked myself – “how do I test this”. I needed something lightweight that would fit in our unit tests. I briefly searched the Internet but could not find anything that would be suitable.  As the old saying goes, necessity is mother of invention, so I came up with my own way of testing the schema. I like it because it contains just 3 small (less than 30 lines total) helper methods, one helper schema and most of the unit tests are just 2-3 lines. The tests actually also helped me come up with a better design I originally had. Note, I don’t know if this is the “right” approach or if it would scale for bigger schemas. I only know that for the schema I had to write it worked fine.

So, let’s say we need to write a schema for Xml files that have a structure like this:

<Settings>
    <ServiceProvider Type="typeName">
      <Setting Name="Setting1" Value="Value1" />
      <Setting Name="Setting2" Value="Value2" />
    </ServiceProvider>

   <Factory Type="typeName">
     <Setting Name="Setting1" Value="Value1" />
     <Setting Name="Setting2" Value="Value2" />
     <Setting Name="Setting3" Value="Value3" />
   </Factory>
 </Settings>

and that both ServiceProvider and Factory elements are optional.

First we need to create a starting schema. For new schemas I usually create a sample Xml file, open it in Visual Studio and use Xml → Create Schema. The schema created by the VS is not really usable but gives me something I can iterate on. The main problem with the generated schema is that all the types are defined inline. This makes it hard to test – ideally we would like to test each type separately. Generating inline types leads to another problem – each element has its own type even if the same element is used repeatedly (let alone cases where the same types are used for different elements or where inheritance is involved). The key to testing a schema is to have simple types. The simpler the type the easier it is to test. Once a type is tested it can be used as a building block to build more complicated types but it won’t require any more comprehensive testing as part of the more complicated type. For the Xml structure above we can identify three types:

  • Setting (for Setting element)
  • ServiceTypeInitializer (a common type for ServiceProvider and Factory elements)
  • Settings (for Settings element)

The problem with unit testing all these types in separation is that the schema itself should not allow any but Settings element as the document element. Fortunately for testing purposes we can create a helper schema that will allow document elements of types that are normally not allowed to be document elements. We will conditionally add this helper schema to the schema set used for validating the input Xml. Why the helper schema needs to be added conditionally? The tested schema should not allow any but the Settings element as the document element. So, when testing the Settings element we must not add the helper schema to the schema set to make sure that this is the only element allowed as the document element. Let’s see how this looks like in practice. Here is the schema created by refactoring the initial schema created by Visual Studio:


<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="Settings" type="Settings_Type" />

  <xs:complexType name="Settings_Type">
    <xs:sequence>
      <xs:element name="ServiceProvider" type="ServiceTypeInitializer_Type" minOccurs="0" maxOccurs="1" />
      <xs:element name="Factory" type="ServiceTypeInitializer_Type" minOccurs="0" maxOccurs="1" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="ServiceTypeInitializer_Type">
    <xs:sequence>
      <xs:element maxOccurs="unbounded" name="Setting" type="Setting_Type" />
    </xs:sequence>
    <xs:attribute name="Type" type="xs:string" use="required" />
  </xs:complexType>

  <xs:complexType name="Setting_Type">
    <xs:attribute name="Name" type="xs:string" use="required" />
    <xs:attribute name="Value" type="xs:string" use="required" />
  </xs:complexType>
</xs:schema>

Now let’s create the helper schema that will allow testing each of the types separately:


<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="Setting" type="Setting_Type" />
  <xs:element name="ServiceTypeInitializer" type="ServiceTypeInitializer_Type" />
</xs:schema>

(In the above schema the Settings element is not present since it’s already allowed at the top level by the other schema). After creating the helper schema we need a function that will validate Xml documents against our schemas:


private static IEnumerable<ValidationEventArgs> RunValidation(string inputXml, bool includeHelperSchema)
{
    var schemaSet = new XmlSchemaSet();
    schemaSet.Add(schemaUnderTest);

    if (includeHelperSchema)
    {
        schemaSet.Add(helperSchema);
    }

    var readerSettings = new XmlReaderSettings()
    {
        Schemas = schemaSet,
        ValidationType = ValidationType.Schema,
        ValidationFlags = XmlSchemaValidationFlags.ReportValidationWarnings,
    };

    var events = new List<ValidationEventArgs>();
    readerSettings.ValidationEventHandler += (s, e) => { events.Add(e); };

    using (var reader = XmlReader.Create(new StringReader(inputXml), readerSettings))
    {
        while (reader.Read())
            ;
    }

    return events;
}

There are two interesting points here. First we need to turn on reporting validation warnings. This is because XmlSchemaSet has a nasty behavior where no error is reported if the document element of the validated Xml document is in different namespace that the targetNamespace of the schema. This may result in accepting documents that are not being validated at all. Turning on reporting warnings is the first step to catch this condition. The second interesting point is that schema validation will throw exceptions for validation errors but not for warnings. Again, to catch the condition where the expected and actual namespaces don’t match we have to set XmlReaderSettings.ValidationEventHandler which will be invoked for both validation errors and warnings. Other than that the method is pretty straightforward – we create an XmlSchemaSet instance and add the schema under test and conditionally the helper schema. Then we create an XmlReaderSettings object and set it up for schema validation. We use the reader settings to create a validating XmlReader. Finally we read the input xml with the validating reader – all errors and warnings are reported by invoking the validation event handler we set.
With the test driver method ready we can start writing test cases. We write test cases for each type starting from “leaf” types (i.e. types that are defined using only pre-defined schema types) moving to more complex types. If a type contains an element of a type that has already been tested we just test that schema accepts an Xml with the simplest child element of that type and, if the type is mandatory, the Xml is rejected if it does not contain the element. If there are multiple elements of the same type we just write test cases to test the type itself and not test cases to test all the possible elements of that type (they will be tested when testing their parent type). If there was a hierarchy we would write test cases for the base type and then test cases just for what was added (or removed – in case of derivation by restriction) in the derived type. The test cases themselves are simple – in most cases a hardcoded minimal Xml document is validated using the validation method we created and we check whether expected errors are reported or that there are no errors for valid Xml documents. Some examples:


[Fact]
public void Schema_accepts_minimal_valid_Xml()
{
    Assert.True(!RunValidation("<Settings />", false).Any());
}

[Fact]
public void Schema_rejects_Setting_Type_without_Name()
{
    var error = 
        RunValidation(@"<Setting Value=""ABC"" />", true)
        .Single();

    Assert.Equal(XmlSeverityType.Error, error.Severity);
    Assert.Equal(
        "The required attribute 'Name' is missing.",
        error.Message);
}

An exemplary test suite using XUnit can be found on my github. The Readme contains details about requirements, setting up the environment, building and running tests. If you just want to see what’s most interesting (i.e. the code) you can find it here

Pawel Kadluczka