Introduction: Making a Datalogger for a Kamstrup Electricity / Gas Meter With Raspberry Pi!

About Raspberry Pi

So finally I received my new Raspberry Pi. This is a small, low power ARM linux board featuring a ethernet adapter, 2 USB ports and even a full-HD HDMI port for a mere $30!. Check out www.raspberrypi.org for more information.

After running XBMC on it (no mpeg2 decoding, lame) and trying it out as a NAS (no SATA interfaces, sloooow) i decided to use it as a combined hardware / software project.

Sparks all around

The energy company here in The Netherlands made the mistake to install a Kamstrup 162 J series meter in my home with an mysterious looking phone jack on it. The phone jack kept intriguing me day after day. A small investigation on the internet indicated it to be a serial interface to get the meter counters and LOTS more information.

This instructable describes all the hardware and software I used to create this interface. The end result will be nice looking graphs on cosm, a free data collecting service found on the internet.

What it will give you

At least you will have fun making this interface. It will teach you about the raspberry pi, compiling a C program, interfacing to your FTDI board. And when things start to work: ultimate satisfaction :^)

And last but not least: it wil make you aware of the footprint you leave on this earth. I wanted to understand how much resources our household uses and in time, reduce the amount of gas and electricity we use.

What you will need

* An electrical meter fitted with a P1 port. In my case this is a Kamstrup 162, other Kamstrups will most likely work in the same manner.
* Linux commandline experience.
* Some basic TTL electrical knowlegde.
* A working Raspberry Pi;
* A FTDI cable or breakout board (as found on Sparkfun);
* A mini-usb cable.


All set?

Let's go!


Chris

PS: Because Raspberry Pi is a long word, I will (lovingly) refer to it as rPi...


Step 1: Preparing the RPi

Enter the Pi

I have fitted the rPi with a 8G class 4 sdcard containing the standard Debian ARM distribution of the raspberry pi foundation.

This can be found at http://www.raspberrypi.org/downloads. Write the default debian distribution to the sdcard using the dd command (patience!) and make sure the device boots correctly. Also check if it runs stable (the voltage of some power sources can be too much off-spec to drive the rPi in a stable manner) and connects to your network correctly.

I like to use SSH all the time to login into the rpi. I have included a sample session of connecting to the rPi using SSH:

----------------------------------------------------------------------------------------------------------------------
chris@chris-xps15:~$ ssh pi@192.168.2.145
pi@192.168.2.145's password:
Linux raspberrypi 3.1.9+ #66 Thu May 17 16:56:20 BST 2012 armv6l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu May 31 20:02:23 2012 from chris-xps15.home
pi@raspberrypi:~$
----------------------------------------------------------------------------------------------------------------------

If you use Windows, Putty is a good alternative to the Linux SSH shell.

Because we are doing some hardcore serial communications, I like to have minicom around. This can be installed by issuing the "apt get install minicom" command. Start it by issuing the "minicom" command. If you are as old as me and have owned one of those pre- internet modems, it should remind you of those wonderful years...

Our rPi is now completely configured to get things going!




Step 2: Connecting the FTDI Cable to the RPi

Freeze! FTDI!

The FTDI cable allows us to use the USB port on the rPi to communicate with an RS232 interface. It will connect to one of the rPi USB ports and the other end will connect to the Kamstrup meter.

Inverting the RXD

As the protocol spoken by the Kamstrup is inverted, We have to configure the FTDI cable to invert the RXD pin. This can be done on a Windows machine (it is possible on Linux but the tooling is extremely flaky) using the FT_PROG tool found here.

Making the physical connection

To successfully connect to the meter, I have used a telephone RJ type cable (which should have 4 wires, please check as some cheaper cables only have 2 inside) which I cut into one 20cm end. 

Strip the end of the cable and make sure the wires are nicely enclosed with soldering tin. This will allow us to stick them in the FTDI board without using a special header. If you want, you can solder the wires to a 4-pin header.

Pin configuration

The pins on the ftdi cable are labelled (at least on mine)
I have numbered the pins on the meter from 1-4. This is viewes when standing right before the meter, 1-4 from left to right. The RJ clip is located at the bottom of the connector.

FTDI       Kamstrup P1
DTR  --->  1
GND  --->  2
RXD  --->  4

When you have made all the connections, the rPi should have the USB cable with the FTDI attached to it and the FTDI is hooked up to the meter as described above.


Step 3: And Now for Some Software...

Reading the data from the meter

I have attached a small C program (cryptically called ft) which will read out the FTDI directly. This program is nothing fancy, it will:

1. Lower the DTR to cause the meter to give us the precious information
2. Read until there is no data for 2 seconds
3. Print everything on the screen

To compile this on the rPi, install "ftdilib-dev" on the rPi using "sudo apt-get install libftdi-dev", copy the tar file to the rPi (under the root account in the /root directory to keep it simple), unpack using tar -xvf datacollector.tar and then issue the "make" command in the /root directory.

If everything goes as planned, the compiler will give us the ft program. If you start this program (enter ./ft and press enter), a block of information is shown like:

---------------------------------------------------------------------------
/KMP5 KA6U005464565461

0-0:96.1.1(352345243543524352345353535353243)
1-0:1.8.1(00913.000*kWh)
1-0:1.8.2(00787.000*kWh)
1-0:2.8.1(00000.000*kWh)
1-0:2.8.2(00000.000*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(0000.25*kW)
1-0:2.7.0(0000.00*kW)
0-0:17.0.0(999*A)
0-0:96.3.10(1)
0-0:96.13.1()
0-0:96.13.0()
0-1:24.1.0(3)
0-1:96.1.0(352345243543524352345353535353243)
0-1:24.3.0(120531230000)(08)(60)(1)(0-1:24.2.1)(m3)
(00752.852)
0-1:24.4.0(1)
!
---------------------------------------------------------------------------

If no correct data is shown, the connections to the energy meter are probably not correct. Check them again! If the ft program bombs out with a message that the device cannot be opened:

- check whether you are running as root user. Accessing the FTDI device requires additional privileges.
- if you have another FTDI version (or from a difference manufacturer) check the two hexadecimal manufacturer / device numbers in the ftdi_usb_open call. Use the lsusb command to determine the right hexadecimal numbers for your FTDI device.
- if the only data shown on the screen is a lot of gibberish, check if the RXD pin has been inverted using the FT_PROG utility on a windows machine. It really needs to be inverted! You can obtain this utility on the FTDI website.

Converting the data

The data will be parsed by a small AWK program (cosm.sh and cosm.awk) to create the data to post to Cosm. The AWK program scans the output of the ft program and picks up the pieces we are interested in. Finally, the data is concerted into a JSON structure which can be directly fed into Cosm.

An example of such a data message:

------------------------------------------------------------------------
{
"version":"1.0.0",
"datastreams":[
 {"id":"+T1", "current_value":"00913.000"},
 {"id":"+T2", "current_value":"00787.000"},
 {"id":"TotalkWh", "current_value":"1700"},
 {"id":"W1", "current_value":"0000.34"},
 {"id":"W2", "current_value":"0000.00"},
 {"id":"Gas", "current_value":"00752.852"}
]}
------------------------------------------------------------------------

Test the AWK script by issuing:

./ft | awk -f cosm.awk

The output should resemble the block of JSON code above.

Tie the previous two steps together

The cosm.sh script will run the two previous steps and finally post the resulting JSON structure to Cosm. Before we can do that, we will have to create a Cosm account, create an API key and a feed.

I will describe those steps in the next step.


Step 4: Preparing Your Cosm Feed

Create a cosm account

Go to http://cosm.com and create an account.

Create an API key

Login to your Cosm account and unfold the account menu (the small diamond left to your account name). Select "Keys". 
Now press the [ + Key ] button. Generate a key with the following properties:

- Use any feed
- All access privileges

Now press "Create" and copy the new API key into a text file for future reference.

Create a feed

Go to the console menu and press [ + Device / feed ]. Select "Something else" as device type.

Use the following properties;

- No I will push data to Cosm
- A descriptive title (something like: Energy usage)

You can leave the tags empty but it is possible to use some keywords like "gas, electricity" and so on.

When the new feed is created, make a note of the feed number. You will need this later!

Edit the shell script "cosm.sh"

Open an editor like nano or vi (which ever you are comfortable with) and change the curl line as follows:

--------------------------------------------------
curl --request PUT \
     --data-binary @/tmp/cosm  \
     --header "X-ApiKey: PLACE YOUR API KEY HERE" \
     --verbose \
     http://api.cosm.com/v2/feeds/FEEDNUMBER
--------------------------------------------------

Save the file and do a test run from the commandline (enter "./cosm.sh").

--------------------------------------------------
root@raspberrypi:~# ./cosm.sh
* About to connect() to api.cosm.com port 80 (#0)
*   Trying 216.52.233.121... connected
* Connected to api.cosm.com (216.52.233.121) port 80 (#0)
> PUT /v2/feeds/61796 HTTP/1.1
> User-Agent: curl/7.21.0 (arm-unknown-linux-gnueabi) libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6
> Host: api.cosm.com
> Accept: */*
> X-ApiKey: Padashgdahgjhgsdjhsfdkjfskjgfkjz0g
> Content-Length: 332
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Date: Sat, 02 Jun 2012 14:10:48 GMT
< Content-Type: application/json; charset=utf-8
< Connection: keep-alive
< X-PachubePurgeCache: t:feeds/61796
< X-Runtime: 104
< Content-Length: 1
< X-Pachube-Logging-Key: logging.j7fIUd67vzlDk5l8e06Cr
< X-PachubeRequestId: 65e5f627e1563206bf39afb8c67679ce47baff53
< Set-Cookie: _pachcore_app_session=BAh7BjLc2Vzc2lvbl9pZCIlN2I5Y2RkMzc5YTk4MWM3MjdiODUzNjQyMjdiNWRjOGU%3D--c5f8be5a6632f29b836146fa3a96ec1e5d167cb1; domain=.cosm.com; path=/; expires=Sat, 16-Jun-2012 14:10:48 GMT; HttpOnly
< Cache-Control: max-age=0
< Vary: Accept-Encoding
<
* Connection #0 to host api.cosm.com left intact
* Closing connection #0
--------------------------------------------------

The script should run without any obvious errors.

Open the cosm website and login. Open the console to see whether any values have been posted.

Cosm is now set up. You have created an API key and have created a new feed where the information is pushed to.

Automating the data logger

Cron is a nice way to schedule the data logger. Issue the crontab -e command. An editor will open with the crontab in it. I use the following schedule:

*/2 * * * * /root/cosm.sh

This will upload every 2 minutes.

Step 5: The End


Hopefully all went well for your version of this project.

A lot more can be created using this basic technique. Thing I want to explore are:

- using a mysql database with a PHP frontend you can easily create period comparisons
- publish aggregated data to feeds like twitter or facebook.

Enjoy!

Chris