How to Interface Digital Displays With Your PC

Intro: How to Interface Digital Displays With Your PC

This instructable shows you how to read information from your temperature gauge, water heater, barbecue temperature, coke machine, laundry timer, etc. from your PC. While most modern sensors today have the ability to interface to PC based systems at lot of older devices do not. They do, however, offer a digital display that can be read by humans. You can use RoboRealm to read these digits from those devices and use that information in new ways on your PC.

The setup requires a camera focused on the device to read with the image being passed back to your desktop/laptop computer for processing. Below is a DLink DCS900 that provides a wireless link to access the image from the temperature panel seen attached to the wall. This instructable is shows how to convert the image into a meaningful number that you can use.

You can download RoboRealm for FREE by going to RoboRealm.

STEP 1: Camera View

Camera View

(image#1)

The first task is to segment the digits from the rest of the scene. Note that there are many ways
to accomplish this task and the specific modules you may use may differ from this approach.

To begin our segmentation we need to remove the lighting issue. To do this we use the Adaptive_Threshold module. Adding this module converts the above image to (image#2)

The parameters of the adaptive thresholding should be set to approximately the width of one of the digits in order to extract out the digit. We found 20 to be an appropriate window size.

One can easily see that the adaptive threshold does a great job of eliminating any lighting issues and highlights the digits from the background.

But you can notice we've still got some cleanup to do ....

STEP 2: Image Cleanup

First off you will notice that within the image the digits are black. In order to process them correctly
going forward we need to be manipulating white objects as RoboRealm considers white pixels as foreground and black ones as background. All the following modules use this assumption as the basis for their processing. To remedy this situation we simply invert the image. Using the negative
module. (image#1)

Now that the digits are in the correct color we can continue by removing all blobs (connected pixel groups) that are smaller than 70 pixels. The choice of 70 is largely arbitrary but it should be a number where most of the smaller insignificant objects are removed. The actual size will naturally depend on your situation. (image#2)

The digits are now appearing more clearly but due to the original low lighting that the image was taken in the border of the digits is not very smooth. To smooth things out a little we use
the Smooth Hull module to round out the digits' shapes. (image#3)

But you will notice that we still have a rather big issue. The digits themselves are broken into two blobs. In order to best match blobs we need to combine the digits into a single blob ...

STEP 3: Merging Blobs

To merge the blobs we need to expand all white parts of the image until they touch with nearby parts. An easy way to do this is with a combination of erode and dilate modules. We first perform a 2 pixel dilation in all directions. (image#1)

This helps to connect the digits but in some situations a thin line still separates the digits. If we continue to use the dilate module we risk the chance of digits being merged into each other. So instead of using the dilate module to dilate in all directions we simply specify the expansion to only happen in the vertical direction. (image#2)

Now that the digits have been merged we reduce them closer to their original size by using the erode module with a 2 pixel erosion.

This reduces the digits down to a more visually appealing size. (image#3)

Once the digits have been segmented from the original image we now need to understand the digit properties as represented by blobs to help eliminate other objects in the image that are not digits. This helps to reduce the clutter in the final stages and provides a more stable result.

You can notice that the digits appear within a square rectangle. This rectangle was initially formed by the white casing in contrast to the lower light LCD part of the panel. We can use this characteristic to better focus on just the digits. To do this filtering we use the Blob Filter module
with a single 'inner' parameter. (image#4)

You will notice that while helpful there are still two triangles left in the image. We could use the blob
filter to further filter based on triangles and invert that selection to remove them but instead we
will use the final step to eliminate those objects.

STEP 4: Shape Matching

To recognize which digit is which, we have to compare the resulting objects with a known database of objects.

This known database is simply a folder that contains images of each of the separate shapes to be recognized. The folder we used contains the following images (see below)

This database of images was generated by cutting the final image generated up to this point and manually separating each digit into its own file. This was done by pressing CTRL-C inside RoboRealm when the appropriate image was seen and pasting the image (CTRL-V) into our favorite paint program to then crop and save each image into its own file. Note that the image filenames were specified as 0.gif, 1.gif, 2.gif, etc. in order to make the image to numeric translation easy.

This database was then trained by the <a href="/help/Shape_Match.php">Shape Matching</a> module which matches each resulting object to a single file in the database. The confidence and size filters within the Shape_Match module were then used to trim out any bad matches such as those against the remaining triangles.

Note that the digits displayed at the top of each green box is actually the filename of the image matched. Since we named the database images with the digit they represent it is easy to translate the image name into the actual number. In order to do the final grouping we need to use a little VBScript to combine the separate objects together to create the final number.

STEP 5: Try It Yourself

You can download RoboRealm for FREE by going to RoboRealm.

You can download the configuration (i.e. RoboFile) we used for this instructable and see videos relating to this instructable at Digital Reader Tutorial

Have a nice day!

The RoboRealm Team.
Vision for Machines

Comments

[Tinker83]

if your equipment has a 7segment display, why not just tap into that, and make a 232 or usb dongle? you could get /real/ complicated and use a PIC and put it on your home network too.

[davidprosser]

I wouldn't be too suprised if it was all integrated onto one chip now, so that might not be possible in some cases i guess.

[Alathald]

I was thinking the same thing, though I'd plug into the binary output (beginning of the circuit that converts binary to 7segment or the even more complicated digital display) rather than the 7segment display. It would make it much easier to read out to the computer. If you want to get real elaborate, you could bypass the switches and buttons to control you thermostat, security system, microwave, refrigerator, etc. and use a serial port to easily control all of those things with your computer. Just be sure that if the bypass circuit is removed (say if the serial plug is removed or computer is off), the circuit goes back to how it was.

[josh92176]

I think that the idea was to demonstrate what their new software was capable of. -josh

[duncant20196]

I like! I'm going to try. You don't have any way to output the data at certain intervals? I'd like to use this to chart my electricity meter on an hourly bases...

[DeusXMachina]

At first I though, oh he just took a webcam and pointed at a display, but that program is really cool! It's amazing that they've pretty much open source recognition software. A few things: - Just a thought, if anyone ran into light contrast issues, you could make a black box for it, and supply your lighting, that way it'll be 100% constant (a lot of cams i know have built in "I am on" LEDs it wouldn't be much of a mod to put in a bright white) - The next step to go with this is integrate it into a system a la SmartHouse and make your house do things for you. (Kitchen! Sudo make me a sandwich!)

[gormly]

you are fostering these kind of comments by replying to them.

[fungus amungus]

Mike - 1) opinions are fine, but are better when backed up with some kind of reason. That's kind of a clause of the "be nice" policy 2) so why can't you ignore this instructable if it offends you so? 3) makes sense since you offer no explanations of your own 4) He never said he was sick of people, just people who make rude comments. Others have found constructive things to say in the comments so if you have nothing to offer this may not be the place for you.

[thearchitect]

This is sweet, thanks for sharing!.. K.

[lilpunk1302]

I was hoping for this to be more of a what to do with the numbers AFTER you've captured them, future instructable, I presume( btw, first person to combine this into some form of a Konfabulator(I still call it that, you yanks call it Yahoo! Widget engine) widget, that shows room temp, gets huge props from me.

But yeah, also, wouldn't it be easier to just crop the image to get rid of the other stuff, or move the camera closer?

Also, I disagree with your statement,

"You might be able to but you'd then have a hard time putting the different pieces together since you'd be matching two parts of each digit at a time. In addition to that the matches would be less accurate since matching the bottom and top part of the shape would be a more generic shape and thus less distinguishable from the each other. Filling the gaps makes each shape much more distinct and thus easier to match."

Since the software is matching the image it sees with a preset list of images, it doesn't matter if the image is in 5000 pieces(but at that point it'd be a little too intricate and the adjustments would mess it up, NOT THE POINT) 5000 pieces or 2, the soft. is still doing the same thing:

10 If X = Y, Print Y
20 else
30 if X = Z, Print Z

You get the point.

[frollard]

Thats a really neat process... If it's just looking for certain shapes in the end, would it not be okay to just leave the 'gaps' in the 7-segment display digits?

[RoboRealm]

You might be able to but you'd then have a hard time putting the different pieces together since you'd be matching two parts of each digit at a time. In addition to that the matches would be less accurate since matching the bottom and top part of the shape would be a more generic shape and thus less distinguishable from the each other. Filling the gaps makes each shape much more distinct and thus easier to match.

[richelton]

So are you saying that your software isn't just taking a 'known region' from the camera image and pattern-matching that area to a bitmap database--which would be useless if the 'known' camera placement moved even slightly such as being bumped--but instead it is leaning to 'read' those digits in the incoming image regardless of size or angle of incidence? If so, THAT is WAY cool. Thanks for the instructable.

[creatordave]

i agree with MadmechanicMike There is not much use for this

[josh92176]

This is sooo cool!

[joejoerowley]

Very Cool! Great instructable!