How to Interface Digital Displays With Your PC

This instructable shows you how to read information from your temperature gauge, water heater, barbecue temperature, coke machine, laundry timer, etc. from your PC. While most modern sensors today have the ability to interface to PC based systems at lot of older devices do not. They do, however, offer a digital display that can be read by humans. You can use RoboRealm to read these digits from those devices and use that information in new ways on your PC.

The setup requires a camera focused on the device to read with the image being passed back to your desktop/laptop computer for processing. Below is a DLink DCS900 that provides a wireless link to access the image from the temperature panel seen attached to the wall. This instructable is shows how to convert the image into a meaningful number that you can use.

You can download RoboRealm for FREE by going to RoboRealm.

Step 1: Camera View

Camera View

(image#1)

The first task is to segment the digits from the rest of the scene. Note that there are many ways
to accomplish this task and the specific modules you may use may differ from this approach.

To begin our segmentation we need to remove the lighting issue. To do this we use the Adaptive_Threshold module. Adding this module converts the above image to (image#2)

The parameters of the adaptive thresholding should be set to approximately the width of one of the digits in order to extract out the digit. We found 20 to be an appropriate window size.

One can easily see that the adaptive threshold does a great job of eliminating any lighting issues and highlights the digits from the background.

But you can notice we've still got some cleanup to do ....

Step 2: Image Cleanup

First off you will notice that within the image the digits are black. In order to process them correctly
going forward we need to be manipulating white objects as RoboRealm considers white pixels as foreground and black ones as background. All the following modules use this assumption as the basis for their processing. To remedy this situation we simply invert the image. Using the negative
module. (image#1)

Now that the digits are in the correct color we can continue by removing all blobs (connected pixel groups) that are smaller than 70 pixels. The choice of 70 is largely arbitrary but it should be a number where most of the smaller insignificant objects are removed. The actual size will naturally depend on your situation. (image#2)

The digits are now appearing more clearly but due to the original low lighting that the image was taken in the border of the digits is not very smooth. To smooth things out a little we use
the Smooth Hull module to round out the digits' shapes. (image#3)

But you will notice that we still have a rather big issue. The digits themselves are broken into two blobs. In order to best match blobs we need to combine the digits into a single blob ...

Step 3: Merging Blobs

To merge the blobs we need to expand all white parts of the image until they touch with nearby parts. An easy way to do this is with a combination of erode and dilate modules. We first perform a 2 pixel dilation in all directions. (image#1)

This helps to connect the digits but in some situations a thin line still separates the digits. If we continue to use the dilate module we risk the chance of digits being merged into each other. So instead of using the dilate module to dilate in all directions we simply specify the expansion to only happen in the vertical direction. (image#2)

Now that the digits have been merged we reduce them closer to their original size by using the erode module with a 2 pixel erosion.

This reduces the digits down to a more visually appealing size. (image#3)

Once the digits have been segmented from the original image we now need to understand the digit properties as represented by blobs to help eliminate other objects in the image that are not digits. This helps to reduce the clutter in the final stages and provides a more stable result.

You can notice that the digits appear within a square rectangle. This rectangle was initially formed by the white casing in contrast to the lower light LCD part of the panel. We can use this characteristic to better focus on just the digits. To do this filtering we use the Blob Filter module
with a single 'inner' parameter. (image#4)

You will notice that while helpful there are still two triangles left in the image. We could use the blob
filter to further filter based on triangles and invert that selection to remove them but instead we
will use the final step to eliminate those objects.

Step 4: Shape Matching

To recognize which digit is which, we have to compare the resulting objects with a known database of objects.

This known database is simply a folder that contains images of each of the separate shapes to be recognized. The folder we used contains the following images (see below)

This database of images was generated by cutting the final image generated up to this point and manually separating each digit into its own file. This was done by pressing CTRL-C inside RoboRealm when the appropriate image was seen and pasting the image (CTRL-V) into our favorite paint program to then crop and save each image into its own file. Note that the image filenames were specified as 0.gif, 1.gif, 2.gif, etc. in order to make the image to numeric translation easy.

This database was then trained by the <a href="/help/Shape_Match.php">Shape Matching</a> module which matches each resulting object to a single file in the database. The confidence and size filters within the Shape_Match module were then used to trim out any bad matches such as those against the remaining triangles.

Note that the digits displayed at the top of each green box is actually the filename of the image matched. Since we named the database images with the digit they represent it is easy to translate the image name into the actual number. In order to do the final grouping we need to use a little VBScript to combine the separate objects together to create the final number.

Step 5: Try It Yourself

You can download RoboRealm for FREE by going to RoboRealm.

You can download the configuration (i.e. RoboFile) we used for this instructable and see videos relating to this instructable at Digital Reader Tutorial

Have a nice day!

The RoboRealm Team.
Vision for Machines