Introduction: Mobile Data-Backup With Raspberry Pi 3

Picture of Mobile Data-Backup With Raspberry Pi 3

When travelling a few years ago, I needed my laptop to check the weather, book train and flight tickets, check emails and much more.

Now I can do all this stuff with my smartphone and carrying a big laptop with power-supply seems a little out-dated. And there is one function that my smartphone can not provide at the moment: Backup all my pictures and videos from several cameras and sources!

But, you don't need a big laptop for that. A small Wifi-Backup-HDD with SD-card slot is enough. There are devices like this out there, but I don't trust them, because they don't have a display, I don't see what is happening and they cost twice as much as a simple 2.5 USB-HDD.

So I set up a Raspberry Pi 3 to do the job and it works great!

You can use as much USB-HDDs as you want and have full control!

And you can even add your own wifi-hotspot to backup your mobile data too!

Let's Start!

Step 1: Hardware Setup

Picture of Hardware Setup

What do you need:

  • Raspberry Pi 3 (2 or the older 1 will work too)
  • USB external HDD ( like 1TB or bigger)
  • a good 5V 2A Power-supply
  • a raspberry Pi housing or a cardboardbox or similar

Optional:

  • Wifi-Dongle for a Raspberry Pi 2 or 1, the 3rd generation has Wifi on board!
  • one to three LEDs with 1k resistors
  • a small piece of breadboard
  • a pushbutton

Step 2: Software Setup

This is the list of software we are going to use for this project:

  • usbmount: for automatic mounting any usb-stick.
  • Raspberry Python GPIO: for controlling the pushbuttons
  • Bash-GPIO: for direct control of the LEDs from within a bash-script.

Optional:

  • hostapd: for creating a wifi-hotspot with your pi.
  • samba: for easy file-sharing with your phone

Step 3: Optional LED-message Board

Picture of Optional LED-message Board

Because I operate the Raspi without a keyboard and a display, it is hard to say when copying is really finished.

Most external HDDs or USB-sticks have a built-in LED that show read or write access, but this is only a weak indicator.

To shut the raspberry down safely, I use a simple pushbutton. This way I don't have to remotely log in via smartphone to issue the shutdown command. This will improve the data integrity a lot! And it only costs a few cents!

So get yourself a small breadboard, a single 20-pin female connector and start soldering the connector to the bread board. (You can also use a dual-row connector, but these are more expensive and the breadboard might get crowded)

I chose to directly connect the LED-Anode to a Raspberry Pin and via a 1k-Ohm resistor to a ground-pin. You should try and measure if 3.3V (what the rapsberry provides) is enough to light up the LED via the 1k-resistor. I measured my LEDs and they took 1.5mA while providing a good brightness.

The Pushbutton also directly connects a pin with a ground pin.

After some initial testing I found out, that starting the rsync command from usbmount directly didn't work well. Therefore I decided to have another pushbutton, solely for the purpose of starting the copy-process.

Step 4: SD-card Setup and Preparation

Picture of SD-card Setup and Preparation

Get the most recent version of raspbian-lite from here:

https://www.raspberrypi.org/downloads/raspbian/

I used raspbian lite, because I don't need a desktop and all the other stuff, that makes the full blown raspbian so big.

Prepare your sd-card with your PC in the way that works for you: https://www.raspberrypi.org/documentation/installation/installing-images/README.md

Then hook up your raspi with a network cable and power it up. If you don't know how to remotely access your Pi with ssh, have a look at some tutorials or use a keyboard and a display on the Pi itself.

The next step should be the initial change of the password and a software update.

To change the password simply start the raspi-config, there you can also change the locales and other stuff:

sudo raspi-config 

To update the software:

sudo apt-get update 
sudo apt-get upgrade

Step 5: Basic Interfacing and GPIO-Setup

Picture of Basic Interfacing and GPIO-Setup

Well, the setup might not be ideal, but I chose to use a combination of direct access to the LEDs from the console or bash-scripts and python-scripts.

With bash-scripts one can very easily handle LEDs, turn them on and off. But you have a problem with handling interrupts.

This is where python comes in handy. But on the other hand, it is hard to perform file-system copy-processes within a python script. Therefore I use the python script to handle the pushbuttons-Interrupt and then call a bash-script to do the job!

The "gpiosetup.sh"-script initializes all LEDs and pushbuttons to be handled by further software.

The shutdownbutton.py handles the shutdown and copy-button and blinks the little green led. So I know, that the python-script is still running.

The little yellow led only shows that the software has boot up and is ready to copy.

The initial gpio-setup could be done by the user pi, while the python script should be run by root, because we need the rights to shutdown the pi. Just have a look at the crontab-entries how to start the job after boot.

Step 6: Connect the USB-HDD to the System

Picture of Connect the USB-HDD to the System

So first of all, we have to connect the usb-hdd to the system. This should be done during boot-up, but if the usb-hdd is not present for any reason, then the boot-process should of course not fail!

There are several good tutorial how to do this, depending on your hdd and the file-system on it:

https://www.instructables.com/id/Using-a-USB-external-hard-drive-with-your-Raspberr/

https://www.modmypi.com/blog/how-to-mount-an-external-hard-drive-on-the-raspberry-pi-raspian

https://jankarres.de/2013/01/raspberry-pi-usb-stick-und-usb-festplatte-einbinden/

In my setup I chose to mount the usb-hdd in the folder: /media/usbhdd/

The line in my fstab looks like this:

UUID=3284-E8C7    /media/usbhdd/    vfat     utf8,uid=pi,gid=pi,auto,noexec,noatime,rw,nofail    0    0

explanation:

utf8: this is to enable german special characters like ä, ü or ö

uid=pi,gid=pi: mount the usb-hdd in the name of pi

noexec prevents any executable to be run from the hdd

nofail prevents the boot-process to fail if the harddisc is not plugged in.

Step 7: Install and Configure Usbmount

Picture of Install and Configure Usbmount

Next step is to install and configure usbmount. Usbmount is a piece of software that automatically mounts any usb-mass-storage device available. You can configure how the drives are mounted and where.

To install it:

sudo apt-get install usbmount

To configure it, you need to edit the config-file:

sudo nano /etc/usbmount/usbmount.conf

I changed two distinctive lines in the config-file, these are:

MOUNTOPTIONS="noexec,nodev,noatime,nodiratime,iocharset=utf8" 

FS_MOUNTOPTIONS="-fstype=vfat,gid=users,dmask=0007,fmask=0117"

The first line is like the configuration in the fstab, it prevents executables to be run from the usb-sticks and the utf8 takes care of special characters in the german language.

The second line allows user to access the mounted usb-sticks.

Step 8: Optional: Install and Configure Wifi-access Point

This step is optional. You only have to do it, if you want to remotely access your pi with your smartphone or tablet. Which is a good thing, because you can log in and have a look, what is already there in the backup. Or check if a specific file is still present or not. And you can even use it to backup your phone!

This step is not a big deal, but there are already a lot of good tutorials out there, so I only refere to some tutorials:

https://frillip.com/using-your-raspberry-pi-3-as-a-wifi-access-point-with-hostapd/

https://cdn-learn.adafruit.com/downloads/pdf/setting-up-a-raspberry-pi-as-a-wifi-access-point.pdf

Step 9: Optional: Install Samba for Easy File Access

Picture of Optional: Install Samba for Easy File Access

Once you have your wifi-access point running, you should think of installing samba. This provides the service you need to access the raspberry pi with a file-manager on your phone.

But why to use samba, when NFS works much faster and is safer?

Well, I found out that using a NFS-share from an Android phone is not that easy. From a windows phone or a PC it might be even harder.

And because this server is not running 24/7 and "normally" not connected to the internet, the samba-service is not that big issue.

Just follow this simple tutorial and adjust to your need: http://www.raspberry-pi-geek.com/howto/Samba-Deploying

By adding additional shares in the smb.conf you can even provide access to the usb-hdd or other folders of the system.

Step 10: The Essential BACKUP-SCRIPT!

Picture of The Essential BACKUP-SCRIPT!

After so many steps of preparation we now come to the most important script of this device: the backup-script itself!

What do I want it to do?
Well, backup each usb-stick into a special directory on the usb-hdd and only copy new files while still keeping the old ones of course.

O.k. a two step approach:

  1. Get the UUID from the usb-stick and use this as a directory name on the hdd for the backup.
  2. Use rsync to do the backup.

And of course a little side-action like switching on and off the LEDs and unmounting the stick.

That is what in principle happens in the script provided by this step. No rocket science, but sophisticated bash scripting triggered by the python-script which handles the pushbutton interrupt.

Step 11: Get Everything Together and Test It!

Picture of Get Everything Together and Test It!

Now that we have everything together and running it is time to test the function:

Start up the raspberry with the usb-hdd plugged in and wait until the little green LED is blinking.

Now insert an usb-stick and press the copy button. Then the red LED should start to light up and the stick and the hdd should start to show some kind of access. While the copy process is still ongoing, when you issue "df" from the console, it should at least contain a line with "/media/usb0". When the red LED is off again and you issue "df", the line with "/media/usb0" should be gone. This indicates that the usb-stick was unmounted correctly. Now go to the hdd-folder and have a look if all the data from the stick was correctly backuped.

If this is the case you are done here and have a very nice small backup-device powered by raspberry pi!

Step 12: Outlook...

Picture of Outlook...

I wished I could have used the Raspberry pi zero or the model A for that project, but unfortunately they only have one accessible USB-Port. And the zero only provides a micro-USB connector.

I could have used a USB-Hub for it, but didn't like the idea of powering an external HDD and a memory-stick with only one port.

In the end I think it turned out quite nice. The Raspberry Pi housing is nearly as big as the external HDD-housing. The cables are not too long and everything could be stored in a small box.

And the housing provides a good protection against unintended shorts or ESD.

In hotels, where there is only a wired connection: I normally unplug the hdd and use the raspberry pi as a simple access point to provide wifi for my phone. That even saves me additional wifi-access point hardware!

This is the backup of a ~150MB file from SD-Card to the external hdd, from boot to shutdown!

Comments

andyk75 (author)2017-06-19

After my first holiday trip with >80GB of video-backup, I am very happy with the device. But I also found one micro-SD card which would not be mounted automatically by USBmount. By looking at the output of "dmesg" I found that this card registered itself as "/dev/sde". I'm not sure why, but I had to manually mount it to /media/usb0 with the command:

mount /dev/sde1 /media/usb0

The backup-process was then started by button as usual.

htrisakti (author)2017-06-18

Thanks for this project.
i just got my first rasp zero W + hub that i want to learn to use for this project.
1) i'm not comfortable with electronics, so I'll skip the LED indicators. what script do i need to modify?
2) without modifying fstab, is it possible to plug in any HDD? I want to use it to plug multiple HDD.
3) is it possible to control the copying from phone app connected via wifi?

andyk75 (author)htrisakti2017-06-19

Hi, congrats to your Pi Zero! :-)

1) you don't have to edit anything. If you don't implement any LEDs or Buttons you just won't see anything and it will be hard to start the copy process, but you can do that also remotely. --> see 3)

2) Of course you can add several HDDs. If you don't modify the fstab you would have to manually use the "mount"-command after plugging in each HDD to use it. But that could also be done via console or remote-console. --> see 3)

3) Yes of course. Get yourself a ssh-shell app like "Juice SSH" or something similar. The you can log into your raspberry pi just as you would do with your PC. Now you can mount the HDD, because it is very likely not mounted by USBmount. And then you can backup the usb-stick by using the "rsync"-command from the copycontent.sh script from step 10.

But of course you would have to adjust the commands to suit your needs. And if you close the connection, then the sync-process might very likely be interrupted!

Try to do one step at a time and do not try to do all at once! :-)

Good Luck and post your result!

GianlucaR11 (author)2017-05-26

Awesome tutorial! =)

I'm a totally noob on Pi, so i'm having a huge doubt on step 5.
These scripts should be uploaded into Pi? How can i do this?

andyk75 (author)GianlucaR112017-05-26

Just google "crontab", "bashscript" and what to do with it!

In short: The Cron-daemon can start scripts for you, at boot-time, or on special times you can define. With the command "crontab -e" you can edit this list.

The first two are the scripts which really do something, the later two are a printout of the crontab, to show you how these scripts are called. If you create a new file, don't forget to set the executable-flag with "chmod +x yournewfile".

GianlucaR11 (author)andyk752017-05-30

Tks so much andyk75!
I miss the flag step.

Also, i spend a long time trying to set the script to run in exfat system.
A few redos, but eventually everything worked out! =)

Che Grillvara (author)2017-05-29

Thanks, I was planning something like this!
However I want to do it with the zero using the zero4u USB hub.
http://www.uugear.com/product/zero4u/

andyk75 (author)Che Grillvara2017-05-29

Hey, cool. I didn't know that! Looks very interesting. Maybe I should order one?

Thanks for sharing

javi_el_largo (author)2017-05-28

Hi, AndyK75, I think it's an amazing instructable. I recently bought a Rpi Zero W which, as you know has built-in wifi, and in a couple of days I'll be buying a Gopro, so this project is perfect for me. I would like to ask you if you have any advice to port it to Rpi Zero W and if you have some kind of schematic for the button and LED board.

Thank you.

andyk75 (author)javi_el_largo2017-05-28

Hi Javi,

I added a small schematic on how to connect the LEDs and buttons to step 3.

My biggest concern with the Pi-Zero W is the power supply. The power it can deliver via the one USB-port is very limited. Some time ago I already had problems running an external keyboard!

Of course you could, or better should use a powered hub to connect the external HDD. And then it would of course be efficient to use this hub also to power the pi. But powering the Pi from a powered hub, which itself is connected as a downstream-hub feels kind of wrong. Did anybody try this?

Would be interested if it works and how you solved it!

javi_el_largo (author)andyk752017-05-29

Hi andyk75,

I didn't see the schematic, thank you. I was thinking of something like this: using the power supply for both the Rpi and the external HDD, and connect only the data lines and common ground between the HDD and the Rpi, like in the picture I´ve sent. No need for powered USB hubs.

I've not tried it yet, but I'll do as soon as I receive the raspberry.

alCHMist (author)2017-05-24

Good tutorial mate another good use for the good old Pi :) I'd like to address the lame comment below by the Galaxy Note fanboy! What has your comment got to do with building your own cheap WiFi backup solution? It doesn't does it mate, yes you can probably do part of the backup with any good recent phone or tablet but that's neither a dedicated solution or as elegant as this one provided in the tutorial. Also I don't know if you didn't notice but you need 2 usb ports for the data transfer? 1 for the source memory/sd card or other storage device and 1 for the the target drive so unless you mean to backup to your phone/tablet storage/memory/sd card which is not the same thing is it or you have 2 OTG ports then your a bit knackered ;)

Peter Sanders (author)2017-05-23

Hi

Great project, well done.

This can be done with a Galaxy Note 3 and an OTG cable :-) Low power HDDs can be powered by the phone (I've tested up to 2gb HDD). Full interaction with ALL mounted drives/memory cards with progress display on the phone. Copying can be done directly between mounted drives, no need to copy to the phone first for later transfer to other device. This ability (except HDD power) has been available since the original Galaxy Note in 2011.

kind regards

Peter

Raphango (author)2017-05-23

Great idea!! =D Thanks!

andyk75 (author)2017-05-23

Thanks for the comments.

Of course you could add a screen!

There are several small screens with or without touch available. The disadvantage is: They need additional power to operate and you need to write additional code to display what you want.

I skipped that. :-)

Aaaecm (author)2017-05-23

Would it be possible to add one of those tiny screens, say 2 x 3, to track the download progress? I am definitely making one of these for my son who takes many pictures when traveling. Thanks for posting!

srilyk (author)Aaaecm2017-05-23

You could do whatever you want - that's the great part about a Raspberry Pi!

gcai_fwb (author)2017-05-23

great idea and good execution - however a much simpler and less expensive solution, if you have a Android based phone is an OTG cable

- see http://thetechieguy.com/an-absolute-must-have-the-...

they cost ~$5+ on amazon (https://www.amazon.com/s/ref=nb_sb_noss_2?url=sear...

or you can make one yourself - see https://www.youtube.com/watch?v=aEo_nJ3iUYI - start @ 1:30 for example

I've used OTG to perform picture backups:

-copy pictures to internal SD card on phone from the camera SD card

-then copy from the internal SD to an external USB stick

-if your internal SD card is big enough you can leave the pictures there giving you 3 copies of your pictures, else delete from the phone

-extra bonus: if you have enough data bandwidth/roaming is to backup in the cloud from the phone

tbrown55 (author)2017-05-23

That's a fantastic idea. I've been trying to rack my brain for a use for a raspberry pi and I might use this project as inspiration. I'm backup mad and the idea of being about to plug in a hard drive to replicate it to another without using up my main system would be super handy. Thanks for the wonderful guide!

RonaldG (author)2017-05-23

Awesome project! Going to try it... than you for sharing! ??

michaelrbellis (author)2017-05-23

This is very cool, dude. Thanks for sharing your great invention. Brilliant Instructable.

JohnP280 (author)2017-05-23

Could you add some sort of counter ?

About This Instructable

27,839views

369favorites

License:

Bio: I like to explore new things and try out stuff. At the moment I'm in to electronics, BLE and LEDs.
More by andyk75:Mobile Data-Backup With Raspberry Pi 3Micro Bluetooth Low Energy (BLE) Robot (for Roborally?)Make 3D-action Videos for a Budget-price
Add instructable to: