Sunday, 11 March 2012

Time for a REST: How ServiceStack solved our mono woes

We were about to abandon our Linux based project due to performance problems and memory leaks with mono ASP.NET and then we discovered ServiceStack.

We're a start-up company of two and prefer developing in Linux using open-source software and libraries. This is also cost effective for the business since we have fewer licences to purchase, both on our own machines and any systems we sell. Additionally, remote management through SSH is a major advantage.

The short version

We have been developing an instrumentation system with a web application front-end. After spending a large amount of time battling memory leaks, stability and performance issues with the mono ASP.NET implementation we finally switched to ServiceStack which has resolved all of those problems.

A generic instrumentation system written in C#

A while back Justen came up with the idea of developing a generic instrumentation system. Many industrial instruments are provided with their own specific software containing poorly designed interfaces or in some cases no software at all.

Our application consists of two parts:
  1. A daemon that is responsible for talking to the instruments.
  2. A browser interface that runs on a kiosk but is also available over the network, thereby allowing engineers to monitor and control individual instruments remotely if required.
The browser interface is mostly implemented in javascript with a C# server-side web application to provide the instrument readings, status and configuration information.
The daemon application is also implemented in C#, since there are libraries available for communicating with serial ports and USB devices, meaning that we could share code with the web application.

The original design: MVC and NHibernate

A database would be used to provide the configuration of channels and instruments, it would also contain information about the instrument specific variables. These variables would be populated with data by the daemon and read by the web application which would also be able to "post" modifications back to the daemon where appropriate (variables that represent instrument settings, for example).
The technologies used in the first design were as follows:
  • Mono (since we wanted to deploy on a Linux based kiosk/server).
  • MySQL database.
  • Fluent + NHibernate libraries to simplify database access.
  • MVC2 as the web application framework.
  • MooTools javascript library.
This design had many issues but probably the biggest flaw at the time was the use of the database to store the transient variable values read from the instruments. Not only was this a major bottleneck but it also caused constant disk access.

The second attempt: MVC3 and memory tables

When it came to the second version we foolishly thought that the way to solve the issue was to use memory tables in the database. This version also involved a major redesign of the javascript engine:
  • Instead of multiple views in the MVC there would be a single view to load the javascript application which would then handle the menu options and various displays itself. The controllers in the MVC would still provide the JSON responses for the various requests made from javascript.
  • We did not fancy having to implement specific javascript classes for each instrument added in the future and so we came up with the crazy idea of describing instrument panels with XML that could then be parsed and rendered appropriately by the javascript (this gives us the opportunity to create the panels in other applications from the same XML).
We moved the web application to MVC3 and after a lot of work produced components capable of rendering themselves from XML (instrumentation specific since it involves binding to instrument variables amongst other things).

Discovering problems by polling ... a lot

The web application is rather unconventional as it involves a long running javascript engine polling the server at regular intervals (every 500ms or so) for both status information and variable readings. This uncovered some rather major memory leaks, especially when left running for days on end. For an instrumentation server this is no good, the application has to be able to run indefinitely without causing the system to eventually grind to a halt!

And thus began the tedious search for leaks...

Memory leaks and stability issues with System.Web under mono

We spent some time tracing the cause of the leaks, although that was nothing compared to the amount of time we've spent trying to prevent them, since they did not appear to originate in our code. The major culprit was traced back to System.Web and we submitted a bug report, but that has been open for over 6 months now. 

A partial workaround was to completely disable sessions but we wanted to add authentication to parts of the application later on and so we also experimented with the MVC3 SessionState attribute which should have allowed us to disable sessions for most of the controllers. However, it would seem that there is some supporting functionality missing in mono so this attribute has no effect. Another bug report was submitted, but at the time of writing this post there had been no response.

We solved the authentication problem at the time by switching to a simple method of our own that stores tokens in a memory table which are referenced by cookies.

Even with a variety of workarounds we were still leaking when running through XSP and using the Boehm garbage collector, but with SGen the leaking seemed to stop. Even with SGen enabled, it still leaked when running through apache and mod_mono, therefore we started running our application through XSP and used mod_proxy to provide access to it.

Unfortunately that wasn't the end of it since the application would randomly crash when using SGen, so we had the choice of a gradual leak with one GC or random crashing with the other. We also suspected that something in NHibernate was leaking, but it was hard to pinpoint. On top of all this, this web application would intermittently not start up properly with errors regarding duplicate keys in the role manager or object null exceptions.

The daemon, on the other hand, shared a lot of code with the web application and yet it seemed stable, was perfectly happy running under Boehm and did not leak memory.

Last year, when we first started discovering all of these problems Justen wrote about them and some of the interim solutions we found

After a lot of time and effort we were about ready to either ditch the project entirely or resort to running it under the .NET framework. This would have been difficult for us to support since we develop in Linux, plus Windows licences would have been required for any kiosk/server units and all our work in setting up a kiosk image would have gone to waste. We would also lose the ability to simply SSH into the units remotely to configure or troubleshoot them when at our distributors.

The silver lining - ServiceStack

The switch to memory tables for the variable readings had stopped the constant disk access but had actually made the performance bottlenecks worse (it seems memory tables have some serious limitations). The solution was, of course, to use a memory caching system like memcached which would allow both our daemon and web application to access the information.

At this point we stumbled across ServiceStack and realised that we had inadvertently developed a RESTful application since the controllers in the MVC were simply acting as services for the javascript engine. Even though we could not really afford to spend more time on the project, we decided to take a week or two to attempt porting our application to ServiceStack and see if that resolved any of our issues.

As soon as we began porting to ServiceStack, things started looking up:
  • ServiceStack provides a caching interface with memcached being one of the available providers.
  • We can run our services through a console application and then simply host our javascript, css and images with apache.
  • Our session-less authentication was simple to port to ServiceStack and has been updated to use the caching system.
  • Start up time of the application is much faster.
  • Very little boilerplate code or configuration is required.
  • Simple to use with lots of helpful examples available.
  • Source code is straightforward to follow if you want to find out how something works.
  • It encourages the use of forward-compatible data transfer objects (DTO).
  • Helpful and friendly support from the developers. 
The only problem we found during the port was nothing to do with ServiceStack but an issue with how browsers handle asynchronous GET requests.

Major performance improvements and increased stability

Switching from memory tables to memcached for the storage of variable readings completely removed the bottleneck, thereby providing us with an order of magnitude increase in speed for those requests.

The entire application was running much faster, mostly thanks to the extremely efficient JSON serializer used by ServiceStack, and the memory leaks were massively reduced but not entirely eliminated. This may confirm our earlier suspicion that NHibernate might be leaking.

Fortunately ServiceStack also provides a very fast and extremely light-weight library called OrmLite. Having had some success with switching our application to ServiceStack we decided to take the plunge and rip out NHibernate, replacing it with OrmLite.

Now, I'm not keen on writing SQL and managed to avoid it completely by using NHibernate, but our database is not very complex and therefore the transition to OrmLite has not been too difficult. In fact, there are certain queries that I've been able to optimise very nicely compared to their NHibernate counterparts, further improving the speed of our service requests.

Leaky results?

Having only just completed the port there is still a fair amount of testing to do, but the results so far are promising. Both the daemon and REST application have been running for a few days now and neither appear to be showing signs of leaking memory!


Mono is a fantastic project bringing C# to all platforms, however we have been very disappointed with the lack of interest in solving what seems to be a rather critical bug in System.Web. Combined with the instability issues and lack of packages for debian/ubuntu we were considering giving up on mono altogether. 

However, the excellent ServiceStack came to the rescue and has provided us with the means to continue developing our applications under mono on the platform we prefer. It's just a shame we didn't discover it earlier!

Friday, 2 September 2011

Custom USB devices on an Android phone

I have a Samsung Galaxy S2 which is a lovely phone that I've had plenty of fun customising thanks to its unlocked bootloader and open-source kernel. One feature of the phone that is of particular interest is the USB On-The-Go (OTG) support which allows it to switch between slave and host modes, meaning that with the right adapter you can plug a mouse, keyboard or even memory stick straight into the phone and it will handle it (as long as the power requirements for the peripheral are very low).

My work involves dealing with a custom USB hardware device and so I was very keen to find out whether or not I could talk to the hardware via my phone.

Powering the device
The first step was to provide external power since the SGS2 would be incapable of supplying enough to run the device. This was simply a case of taking an old USB cable and attaching the +5V and ground wires directly to the supply pins of the device so that I can power it via a computer or phone charger (a final product could involve some form of battery instead).

Debugging the phone via adb
I also needed a way to access the logs and shell on the phone which would normally be achieved using the USB cable and adb, but since the port was going to be used for the OTG adapter this was not an option. Instead I found that it is possible to connect to adb over TCP/IP, free widgets can be found in the market that will enable it using a custom port number (5555 is the default) and then you can connect to the device using:

adb connect

Initial testing
Having already confirmed that my OTG host adapter was working by testing a mouse and memory stick, I then started experimenting with plugging our device into the phone. I had noticed that when supported peripherals were plugged in it was logged in the standard logcat (via adb) but nothing appeared for our custom hardware. Therefore I started to dig a bit deeper and monitored the dmesg output only to find that it was noticing the device but then rejecting it and reporting that "device vXXX pYYY is not supported" where XXX was the vendor ID and YYY was the product ID.

It was time to start digging through the kernel sources (initially pulled from the repository set up by supercurio), which Samsung have very sensibly open-sourced, looking for the log message above. I tracked it down to some header files in "drivers/usb/core" that were named:
  • otg_whitelist.h
  • s3cotg_whitelist.h
  • sec_whitelist.h
So it would seem that Samsung/Android have set up whitelists of devices that are allowed to connect (primarily mass storage and HIDs), how annoying!

Building a kernel
The next big challenge was to figure out how to build my own Android kernel. It proved quite tricky finding all of the relevant information although a post by codeworkx helped a lot. At the time of writing this post the kernel sources repo was unfortunately out of date and so I switched to the one listed in the post. I also pulled the Android sources for the build toolchain and began building.

A number of soft-bricks later I managed to get a self-built kernel running successfully on my phone (including an initramfs in the kernel build is an easy thing to miss but oh so important). It was scary to test but very satisfying once working. Thank you Samsung for your wonderful download mode without which I probably would have completely bricked my phone! Thank you also to Benjamin Dobell for his Heimdall application which now fully supports the SGS2 and provides us linux users with a method of flashing our phones in download mode.

Hacking a kernel
Now I could really get down to business and started hacking those header files I found earlier. I chose the obvious one (s3cotg_whitelist.h) since it seems to refer to the Samsung "Systems-on-Chip" and added the following to the whitelist table:

{ USB_DEVICE_INFO(0xff, 0x0, 0x0) },   /* vendor specific USB devices */

After rebuilding the kernel and flashing it I still found that the custom device was being rejected. It seems that the important file is actually "sec_whitelist.h" in which there are two tables, so I added my little hack to both, rebuilt, flashed....success! The device was accepted and the connection was even visible in logcat.

Since then I have also discovered that whitelisting is a kernel option. I have not tested it but would assume that if you edit the kernel ".config" file and set "CONFIG_USB_SEC_WHITELIST=n" then it would allow any type of USB device to connect.

Writing an application
Now that the device can connect to the phone it is time to start developing an application to communicate with it. The Android SDK provides a number of classes that give direct access to USB devices and if you have experience with libraries such as libusb then it will seem fairly familiar.

One thing to note is that the UsbDevice and related classes are part of Android 3.1 and so an application needs to be built against API level 12 but to run it on the SGS2 you can get away with setting the minSdkVersion to 10 in the manifest (this will generate a warning but it's all good). 

Thursday, 21 April 2011

Adaptive Threshold Edge Detection in GIMP

We have a particular edge detection algorithm that was developed for the OpenIllusionist project specifically to help with detecting fiducials in live video since it was faster and better suited to the task than algorithms such as the canny edge detector.

The original algorithm was designed by Prof. John Robinson at the University of York and then modified and optimised by myself. The following diagram, taken from my thesis, provides an overview of how the algorithm works:

The GIMP is an extremely useful tool, not only for manipulating your photos but also for prototyping machine vision algorithms. Since the edge detection is part of our standard toolkit it would make sense to be able to test it out on images alongside all of the usual filters found in the GIMP.

The porting of this algorithm to a GIMP plugin was not simple, since GIMP plugins usually make use of tile-based processing thereby reducing the overall memory required to handle very large images. However, as can be seen in the above diagram, the adaptive threshold edge detector uses many full and scaled down buffers to produce the final result and as such cannot easily be modified to work with tiles. The current implementation of my plugin simply uses full image regions and therefore the user must be aware that it may have issues if applied to very large images.

One problem that this particular algorithm attempts to address is that of edge ringing in images, especially those captured from low-quality cameras or affected by compression artefacts. The following image is based on an example from wikipedia but also includes the results from our edge detector which clearly shows that it is unaffected by the jpeg compression artefacts present in the lossy image:

The code for the plugin along with a binary for Linux x64 can be found in my github plugins repository and further examples of the algorithm in action are shown below.

Source image copyright Dan Parnham

Source image copyright Benh Lieu Song

Sunday, 20 March 2011

Cross-compilation Adventures

At work we were recently asked to produce a Windows 64-bit build of our library, however our existing build system is not able to build 64-bit DLLs. The server is a Debian x64 machine running trac, subversion and hudson.

The automated build process is based on a combination of nant and premake scripts (we use premake to generate code::blocks project files for our development machines and makefiles for the build machine). The scripts use gcc/g++ to build the Linux version of the library (.so) for our kiosks and they use MinGW to cross-compile the library for Windows x86.

MinGW is just the standard version installed from the repositories and is not currently capable of producing 64-bit binaries, so we needed to switch that section of our toolchain.

The rest of this post documents how we configured a new toolchain capable of 64-bit builds and makes use of the MinGW-w64 project.

Install MinGW-w64

The MinGW-w64 project provides two toolchains, one targetting Win32 builds and one targetting Win64 builds.

Download the Win32 toolchain and extract to /opt/mw32
Download the Win64 toolchain and extract to /opt/mw64

In /usr/local/bin create the following helper scripts (this is optional but we found them useful):

mw32-ar :

i686-w64-mingw32-ar $@

mw32-gcc :

i686-w64-mingw32-gcc -m32 $@

mw32-g++ :

i686-w64-mingw32-g++ -m32 $@

mw64-ar :

x86_64-w64-mingw32-ar $@

mw64-gcc :

x86_64-w64-mingw32-gcc -m64 $@

mw64-g++ :

x86_64-w64-mingw32-g++ -m64 $@

Building wxWidgets

Our library is built using wxWidgets since it is fast, easy to use and provides cross-platform access to string functions, file handling, XML parsing, image loading/saving and so on. We statically link wxWidgets and so the following describes how to build 32-bit and 64-bit static versions of wxWidgets.

Download wxWidgets >= 2.9 since 2.8.x does not seem to build to 64-bit successfully. As of writing there is also a problem with legacy references that break the build and the simplest way to fix this is to hack the "configure" and "" scripts removing "-lwctl3d32".

Open a shell in the wxWidgets root folder (having extracted it somewhere) and run the following. I recommend going away and having a nice cup of tea while the make is running since it will take a while.

$ PATH=/opt/mw32/bin/:$PATH ./configure prefix=/opt/mw32/mingw --host=i686-w64-mingw32 --enable-unicode --disable-monolithic --disable-shared --build=`./config.guess`
$ PATH=/opt/mw32/bin/:$PATH make
$ sudo make install

$ make clean

$ PATH=/opt/mw64/bin/:$PATH ./configure prefix=/opt/mw64/mingw --host=x86_64-w64-mingw32 --enable-unicode --disable-monolithic --disable-shared --build=`./config.guess`
$ PATH=/opt/mw64/bin/:$PATH make
$ sudo make install

As of writing the make install does not copy the headers to a nice "wx" folder and so the simplest solution is to symlink them:
$ cd /opt/mw32/mingw/include
$ sudo ln -s wx-2.9/wx ./
$ cd /opt/mw64/mingw/include
$ sudo ln -s wx-2.9/wx ./

Since our server is also used to build Linux versions of the library, it has the wxWidgets developmental packages installed from the repositories and therefore also has the "wx-config" tool available. To make this tool aware of our cross-compilation configurations we simply need to symlink the scripts to the appropriate location:
$ cd /usr/lib/wx/config
$ sudo ln -s /opt/mw32/mingw/lib/wx/config/i686-w64-mingw32-msw-unicode-static-2.9 ./
$ sudo ln -s /opt/mw64/mingw/lib/wx/config/x86_64-w64-mingw32-msw-unicode-static-2.9 ./


Building something using the appropriate wxWidgets libraries is now a case of passing the appropriate wx-config command to the compiler:

32-bit build options:
`wx-config --host=i686-w64-mingw32 --version=2.9 --cxxflags`
32-bit link options:
-static-libstdc++ -static-libgcc `wx-config --host=i686-w64-mingw32 --version=2.9 --libs`
64-bit build options:
`wx-config --host=x86_64-w64-mingw32 --version=2.9 --cxxflags`
64-bit link options:
-static-libstdc++ -static-libgcc `wx-config --host=x86_64-w64-mingw32 --version=2.9 --libs`

If this is configured in a makefile then the helper scripts can be used to tell make which toolchain to use, e.g.:
$ make config=cross CC=mw32-gcc CXX=mw32-g++ AR=mw32-ar

Congratulations, you should now have a Linux system capable of cross-compiling to both 32-bit and 64-bit Windows binaries.


I have since found that when setting up a build (using tools such as premake) it is simpler if the helper scripts act in the same way as the native build tools and handle the -m32 and -m64 flags appropriately. The following is an example of the mw-g++ script I am now using to replace the previously defined mw32-g++ and mw64-g++ :


for arg in $*; do
        if [ "$arg" = "-m64" ]; then

if [ "$target" = "x64" ]; then
        x86_64-w64-mingw32-g++ $@
        i686-w64-mingw32-g++ $@

Saturday, 19 February 2011

Nested Elements in MooTools

When developing a web application my usual javascript library of choice is MooTools since it is simple to use, and provides cross-browser support along with a form of object-orientation.

The library has an Element structure that allows for the easy construction of HTML elements, but the way those elements are tied together is based on functions such as inject(), adopt() and grab(). This works well, but I was recently constructing a fake dialog box consisting of many nested elements and wanted to be able to declare them in a way that clearly shows the hierarchy.

This would be one way of doing it with MooTools:
var dialog       = new Element('div.background');
var container    = new Element('div.container').inject(dialog);
var window       = new Element('div.window').inject(container);
var header       = new Element('h1', { text: 'A title' }).inject(window);
var content      = new Element('div.content').inject(window);
var bar          = new Element('').inject(window);
var okButton     = new Element('button', { text: 'OK', events: { ... }}).inject(bar);
var cancelButton = new Element('button', { text: 'Cancel', events: { ... }}).inject(bar);

However, although it is fairly tidy and concise, it is difficult to just look at the code and determine the resulting structure. Instead I wanted to be able to do this:
new Element('div.background', { children: [
    new Element('div.container', { children: [
        new Element('div.window', { children: [
            new Element('h1', { text: 'A title' }),
            new Element('div.content'),
            new Element('', { children: [
                new Element('button', { text: 'OK', events: { ... }}),
                new Element('button', { text: 'Cancel', events: { ... }})
using a property that would allow me to declare child elements.

One of the most powerful features of MooTools is that it provides methods to extend the toolkit further. I have already taken advantage of this in the past by adding a search function to the Array class that takes a predicate function. The Element structure, however, is not an actual MooTools class and therefore cannot be easily extended. I did find an example that created a new utility function which allows child elements to be specified, but I wanted to simply enhance the existing Element.

The best way of enhancing Element is to add to the global element property set as follows, then the corresponding get/set in the Element structure simply use that property when they find it:
Element.Properties.children = 
    get: function()      { return this.getChildren(); },
    set: function(value) { this.adopt(value);         } 

Friday, 7 January 2011

Kindle 1-Click Security

What would happen if your Kindle was lost or stolen while away?

Your Kindle is linked to your Amazon account and you have to enable 1-click ordering to be able to buy books on it. That means if somebody got hold of your device they could potentially purchase a huge number of books before you could even get online to deregister it.

There is the option to add password security to your Kindle but that would mean having to type your password in each time you wake it up. An alternative is to remove all payment details from the account but this would be inconvenient next time you actually wanted to purchase something. I stumbled across a suggestion on a forum and thought I would post it here since it seems to work:

In your "Manage Your Kindle" account page edit your "Default 1-click Payment Method" and add a new card as follows:

Card Number
4111 1111 1111 1111Standard dummy VISA number
DummyOr something else to remind you that this is not a valid card
Start Date
01/11Or any date in the recent past
Expiry Date
01/15Or any date in the future

Before going away you can switch to this card so that if your device does go missing you will not end up with a credit card bill full of book purchases. Using the dummy card does not appear to prevent you from ordering free books through the 1-click system either!

Monday, 22 November 2010

Invisible Starfield

After publishing Starfield Twist over the weekend I was unable to see it in the market on my Tattoo. A couple of releases later and it's finally there and all it took was the removal of a bit of XML. It seems some phones, that are perfectly capable of using live wallpapers, do not declare it correctly (including the unofficial 2.2 ROM I am running) and so the market automatically filters out those apps.

The fix is to remove the following from the XML which tells the market that the app is a live wallpaper:
<uses-feature android:name="" />
The downside to this is that all devices running Android 2.1 and above will see the app even if they do not support live wallpapers!