Introduction: Under $8 Arduino Serial Data Logger - Record to SD Card
THIS IS AN ADVANCED PROJECT - NO HANDHOLDING
You will need access to an AVR programmer, either a dedicated one such as USB-AVR or an Arduino programming shield such as sold by AdaFruit or a home-made programming interface. This will be necessary for programming the naked 328P-PU which will be used in this project. The assumption is that you will be using the Arduino 1.0.1 or higher environment to program the uC. There are several decent Instructables on how to program the ATmel ATmega328P-PU. Please read and research.
Summary of operation:
The sketch makes use of the SD Arduino library. There are 3 physical connections: Gnd, +5V, and RS232 ASCII signaling at 9600 BAUD. The 328P will buffer incoming characters until the buffer is full (512 characters) or until the datastream has a Carriage Return (CR) at which time the data in the buffer is written to the SD Card. I have tested 1G and 2G inexpensive cards formatted FAT with no issues noted. That is all this project does... just records serial data onto SD.
Parts you will likely need to purchase:
- ATmel328P-PU blank chip - approx $4
- SD Card holder such as LC Studio for approx $3.50 from eBay
- 16MHz (20ppm or 30ppm) crystal for approx $0.50 from any Internet seller
- Small load capacitors, 2x, of approx 18-22 pF to match crystal requirements
- 10uF to 100uF bulk capacitor rating 10V or greater
- 10K resistor / 22K resistor / 150 Ohm resistor all 1/4W
- 100pF, 2x, ceramic bypass capacitors for power lines
- Tri-color LED or separate Red, Green, Blue LEDs (Optional but useful)
- Screw-down lead connector (optional)
- 28 pin IC socket (optional, but highly recommended)
- Small prototype board - shown is from Radio Shack
- Misc. lengths of hook-up insulated 24G wire
- Soldering iron, solder, hand tools, and other things I forgot to list
Why you may want to build this project:
Because this self-contained project will simply log anything that is thrown to it at 9600BAUD from a serial output of another device. In software, you can lower the serial speed to 4800 BAUD or lower if necessary. You may have success with serial rates above 9600 BAUD - but I have not tested this as my need is strictly selfish and centered around 9600.
Yes, this device can read and log PC terminal output.
Step 1: Overview: Off-loading Serial Data Recording to a Dedicated Microcontroller
Before I get into specifics, a little background may be useful. You've already read the disclaimer that there is no guarantee that this will work, but let me explain why it will provided the ATmel chip you get is a prime chip (I order in 25 QTY from Newark, so I get the microcontrollers for about $2 each and I am assured of prime chips. I also order the 16MHz crystals from Newark.) Although there is no guarantee, there is a very high chance of success when you use quality components.
There are many SD card shields from forces such as SparkFun and AdaFruit. These boards are designed to fit onto your existing UNO Arduion and provide the electrical interface and physical card holder for the SD Card. However, while convenient, this "may" be a poor design choice from the software point-of-view. Primarily, the <SD.h> Arduino library is a RAM hog, consuming 25% of the memory of the device immediately. And, if the programmer feels a need to further buffer, well there goes more RAM. So, as the user's programming logic becomes more complex that just reading a thermometer, the stress on the uC becomes higher. Unless very careful planning and design are utilized, the resources of the Arduino UNO will be stretched too thin.
I am working with an ATmega2560 (Mega2560) and I'm processing multiple engine sensors on an experimental Europa aircraft fitted with a Rotax engine. The ATmega2560 has the additional RAM to support the SD Card buffer, but I'm using both the SPI and I2C interfaces already for other critical sensors. I could extend the chip-enable logic to support more SPI devices, but I really did not want to do this.
Instead, I started looking for other solutions for logging the 9600 BAUD ASCII stream log generated every second as the 2 OLED 2-line displays were updated. The airplane owner has a separate navigational system that provides critical logging, so the engine logging is considered non-critical; a nice to have log of engine performance and such readings as fuel level, ambient air temperature inside the cabin, external air temperature, and so forth. Therefore, an expensive solution from the purchase of a pre-built recorder was not going to sit well with my friend. I needed to be creative.
Way back in time before the first line of code was written, a long list of possible components was purchased from Hong Kong. Real Time Clock modules (see my writeup: https://www.instructables.com/id/DS1307-Mental-Health-for-Arduino-Users/) and buffer chips, and pin headers, and somewhere in the box a couple of LC Studio full-size SD Card holders breakout boards. Ah, Ha... something that was already in the workshop.
As I did my dutiful research on the Internet regarding the LC Studio SD Card, I read article and article about failures; there were very few successes. I dug around and found the schematic for the breakout board. Ummm, I said to myself and maybe out loud, too. The little $2 board had a low-dropout 3.3V regulator. Things were getting interesting. My previous experience with SD cards involved specialized IC to do level shifting. In my mind, I was thinking that IF I COULD run a solo-328P at 3.3V then there would be no level shifting needed. My idea was to simply supply the 5V to the SD-Card breakout board and pass the 3.3V back to the 328P. For level shifting the 5V serial output from the ATmega2560, I would use a simple 22K/10K resistor network as is done with the PICAXE RS232 download circuit.
Armed with an idea and enough information to start prototyping, I gathered the required parts and stuck them into my solderless breadboard. I used the Arduino 1.0.1 SD library and the SD example file CardInfo to test the prototype. I used a "real" UNO to string SPI control signals to the LC board and an OLD SACRIFICIAL SD CARD since I was running the signals at 5V!!! But, the sketch worked, the SD card survived, and I was ready to program a naked chip and remove the UNO from the testing. This move would also allow me to test running the 328P at 3.3V from the LC board 3.3V linear regulator with 5V being supplied by my lab supply.
And it worked.
The next step was to write a sketch specific for the Europa datalogger, move the solderless breadboard to a soldered protobard and throw some serious streaming data at the card. For the data simulation, I used my PICAXE QBF generator: https://www.instructables.com/id/QBF-Quick-Brown-Fox-Serial-Test-Generator/
And if failed... but not miserably. Review of the software determined that I had left "Serial.Print" diagnostic lines in the code. My suspicion was that was causing major issues. I removed the offending commands and recompiled.
And it worked.
Step 2: Arduino Serial Data Logger - Hardware & Software
Looking at the bottom of the board, you can see the RED wire is the 5V power input. This wire goes to the LC module 5V connection. The 3.3V output of the regulator then goes to the breadboard +3.3V bus ... runs across on the top horizontal rail. The lower horizontal rail is GND and you can see the two 100pF bypass capacitors soldered to the rail... one at the far left end and one at the 328P power pins. The single yellow wire is the serial input to the uC for recording (needs to be externally conditioned by the 2 resistor network which is not on the breadboard.)
The four parallel'ish yellow wires are SPI control signals. MISO, SCK, MOSI, CS.
SD card attached to ATmel 328P-PU SPI bus as follows:
MOSI - pin 11
MISO - pin 12
CLK - pin 13
CS - pin 10
If you are unfamiliar with barebones Arduinos, this should help: https://www.instructables.com/id/Standalone-Arduino-ATMega-chip-on-breadboard/ I am not doing an Instructable on barebones, so you may have to do some independent research! Oh, yea, there is that disclosure at the first of this article stating that this may not work for you.
YES, Colors mean something:
I use a single tri-color LED with a single 150 Ohm resistor for current limiting. I never have more than one segment on at a time... that means that I turn OFF any ON led before turning ON another one... pretty simple. At power-on, each LED is turned on in sequence. Then, blinking RED indicates a problem with mounting the SD card file system. After 5 seconds, the software resets and will try and re-mount. This gives the user time to get the SD Card and insert before RESET.
GREEN is used to indicate characters being loaded into the program buffer. It is pretty dim because the event lasts but a short time.
BLUE is used to indicate characters being transferred from the program buffer to the SD 512 dedicated RAM buffer and then to the SD Card for sector updating. This happens fairly quickly, quickly enough that the chip will maintain full-sync with a 9600 BAUD input signal without loosing characters.
About SD Cards:
Some forum folks indicate that the Arduino and SD library works better with lower-cost, older SD cards. I can not confirm this as I only have old full-size SD cards and I'm too cheap to go out and buy the new high-speed SD cards just to test. This one is for you...
Removing the SD card will result in lost data, that is, what is in the 328 RAM buffer. Removing the SD Card during a write as the buffer is being transferred to the SD card will result in data loss AND may damage the file system. In general use, only the program buffer is lost (each carriage returns dumps the buffer to the card) so the correct process is to remove power and then remove the SD card... a minimum of loss will occur. Complete loss can be avoided by having the upstream data stream program issue two (2) carriage returns... the first one will write the buffers and the second one will simply write a blank line. This however does create extra "wear" on the SD card. Another solution is to use any of the many unused digital pins and install a push button and write a little code to force a buffer dump when the button is pushed. Lots of options which I did not implement.
This is probably the strangest Instructable that I have authored in my two years here... because the project is not intended for beginners or intermediates... I'm just not providing that level of instruction/support. I hope if you take on this project that you are successful... I was.
Guess there is nothing else to do than to upload the ZIP source code.
We have a be nice policy.
Please be positive and constructive.