Introduction: How to BackUp and Shrink Your Raspberry Pi Image

Foreword

So why all the hullabaloo you may ask?

Surely all I need to do is use Win32DiskImager and take a read of my Raspbian Image and zip it right?

Job Done!

Well, unfortunately that's not the whole story. Read on...

  • Have you ever noticed the size of your Rasbian image is that of the card you took the copy from and does not reflect the size actually being used?
    • This happens when you re-size your image (via raspi-config) to make use of the full uSD card even if you end up not using much of the card space, the saved image will still be the same size as the uSD card.
  • Have you ever found Win32DiskImager fails to write a backup Raspbian image even when you are writing to a 'like for like' uSD card of the same stated size, just a different make?
    • This is because not all card sizes are the same ie. a 16GB uSD card from one manufacturer may not be exactly the same size as from another. This will cause Win32DiskImager to fail.
  • Did you want to downsize your Raspbian image from one card to another but can't?
    • If you use the Win32DiskImager read method you can only write to the exact same or bigger card size.

The following text details how you can make more efficient backups of your images.

Introduction

As mentioned above this Instructable documents how to efficiently make backups of your critical Raspberry Pi images, compress them such that they occupy the least possible space and can potentially be transferred to a uSD card of smaller size.

The Instructable will cover the following topics;

  1. How to install prerequisite software on Raspberry Pi to allow image resizing.
  2. How to read and store an Image from a uSD card (via Linux command line and with Win32DiskImager),
  3. How to create your compressed backup image,
  4. How to write your backup image to a uSD card (via Linux command line only. it assumes you know how to use Win32DiskImager to write an image. If not go here https://www.raspberrypi.org/documentation/installa...),
  5. Summary,
  6. References used.

What parts do I need?

To set the system up such that you can successfully carry out the instructions you will need the following;

  1. Raspberry Pi 3 + Jessie Image installed (in this case I used : 2016-03-18-raspbian-jessie.img) with monitor, mouse and keyboard etc.,
  2. uSD USB card reader for your uSD card to be backed up,
  3. External USB HDD for Raspberry Pi (big enough to hold your backed up images, in this instance mine is labeled '1TBFREECOM' and oddly enough is 1TB in size, which is more than ample for the needs of this Instructable),
  4. Target uSD card to read the uncompressed Image for backup,
  5. Target uSD card to write the compressed Image to (could be the same as 4.).

.

What skills do I need?

Do I require any specialist skills?

No, only the following;

  1. Some knowledge of Linux is always useful,
  2. A little patience as the writing process may take a while,
  3. The ability to accurately and blindly follow written instruction.

.

Note

    1. This method is only good for Raspbian images having two partitions, so won't work with NOOBS, RetroPi etc.
    2. It is assumed that all Linux commands are carried out directly on the Raspberry Pi (not via a PuTTY SSH connection etc.).
    3. Linux commands are entered via a Terminal instance. Terminal is the 'black monitor' icon on the start bar. Alternatively, it can be found via; Menu -> Accessories -> Terminal.
    4. Although uSD (micro SD) is used throughout, this method applies equally to SD cards.
    5. This was written to complement my series on home automation https://www.instructables.com/id/IoT-Intranet-of-Th...

    .

    Disclaimer

    As always, you use these instructions at your own risk.

    Step 1: Initial Raspberry Pi Software Install

    Before we can begin we must install the following items of software. This is a 'one off' activity and won't take long.

    Enter the following Linux commands, answering 'y' to any questions;

    .

    • sudo apt-get update
    • sudo apt-get install dcfldd
    • sudo apt-get install gparted

    .

    Once complete you have now installed the software necessary to write an image and change the partition of an image.

    Step 2: Reading Your Raspberry Pi Image

    Via Win32DiskImager

    As in picture one above;

    1. Insert your uSD card into your PC USB reader and note which drive it is mapped to. In the example above it is drive 'G:\',
    2. Enter the directory where you want the Image to be written and give it a name. Here : 'c:\temp\RaspberryPi\BackUpServerUncompressed.img',
    3. Click the read button. Circled in red,
    4. Let the software complete,
    5. The Image file located in 'c:\temp\RaspberryPi\' can now be moved to the 1TB FREECOM External HDD for transfer to the Raspberry Pi.

    .

    That's it you're done.

    .

    or Via the Linux command line

    This is a little bit more involved.

    Insert the Raspberry Pi Image to be copied into your USB uSD card reader of your Raspberry Pi.

    Insert your USB External HDD into your Raspberry Pi.

    Open a Terminal instance and enter the following Linux command;

    • df -h

    This is to determine how your uSD card has been mounted (connected/mapped).

    See Pic 2 above for the result. In general you will find external devices have a path something like '/media/pi/...'

    In this case;

    • For the External HDD (named '1TBFREECOM') it is '/media/pi/1TBFREECOM'
    • For the Raspberry Pi Image there will be two entries under '/media/pi/...'
      • One named 'boot'
      • The second will have a long number, here 'e6e7f776-...-c44ecdbfcf90'

    For these two Raspberry Pi entries note the mount details. Ringed in red, here; '/dev/sdb1' and '/dev/sdb2'

    .

    Next we 'unmount', the Raspberry Pi uSD card with the following Linux command (see pic 3).

    • sudo umount /dev/sdb1 /dev/sdb2

    Now we make a backup copy of the Raspberry Pi image to our external HDD, named '1TBFREECOM'

    • sudo dcfldd if=/dev/sdb of=/media/pi/1TBFREECOM/MyImage.img

    Pic 4 shows the backup activity in action and Pic 5 shows what it looks like when it's complete.

    Next we use the sync command to force a synchronise of any outstanding input or output (see pic 6).

    • sudo sync

    Finally we finish off by taking a look at the backed up image file on the 1TBFREECOM HDD to check to see that user name and group are 'pi'.(see pic 7 above. User and Group are circled in red).

    • ls -l /media/pi/1TBFREECOM

    If not, then enter the following command (see pic 8);

    • sudo chown pi:pi /media/pi/1TBFREECOM/MyImage.img

    .

    And that's all there is to it!

    Ok, so I admit it's a little on the tortuous side.

    .
    .

    Gotchas

    In case your image is sizable, you have to format your USB-FDD/-stick as NTFS (example: image of 16 GB on a 32 GB stick). Linux/Jessie distros have a NTFS-driver, which is read-only. Therefore you need to install the newer driver NTFS-3G, which is read & write capable. sudo apt-get install ntfs-3g and reboot.

    Thanks to gorgonops for spotting this. See in line comments below for thread.

    Step 3: Resizing Your Image

    Ok, we'll begin by getting the start point of the second partition on the uSD card. A Raspberry Pi image has two partitions, 1 Boot, 2 Where everything else is stored. To determine the start of the second partition enter the command below;

    • sudo fdisk -l /media/pi/1TBFREECOM/MyImage.img

    Take note of the 'Start' of the second partition '.img2', we'll need it later. Here : 131072 (see pic 1 above, red circle).

    Now we mount this partition using the following command (see pic 2 above);

    • sudo losetup /dev/loop0 /media/pi/1TBFREECOM/MyImage.img -o $((START*512))

    Where START is the start of MyImage.img2 the second partition. ie. 131072. So in this case it will be;

    • sudo losetup /dev/loop0 /media/pi/1TBFREECOM/MyImage.img -o $((131072*512))

    Note : Don't worry if you get a busy message, just enter the following and retry the above;

    • sudo losetup -d /dev/loop0

    Now we resize the partition with 'GParted'. From a terminal window enter the following command (see pic 3 above);

    • sudo gparted /dev/loop0

    This will launch the 'GParted' application (see pic 4 above).

    From GParted do the following;

    1. Select /dev/loop0 partition (pic 5 above),
    2. Select Partition menu -> Resize/Move (pic 6 above),
    3. Resize the partition by either dragging the slider left as indicated, or entering a value into the box marked 'New size (MiB)' (again as shown in pic 6). I found this to be a little trial and error and generally started with +500M and moved up in increments. ie Change the value of "New size (MiB)" to approx 500MB above "Minimum Size". In this case 2942 + 500MB = 3442MB etc.,
    4. Click 'Resize/Move' to finish,
    5. GParted will now indicate there is '1 operation pending' (as in pic 7 above),
    6. Click Edit menu -> Apply All Operations,
    7. Click 'Apply' to warning box (pic 8 above),
    8. GParted will now process the request to resize (see pic 9),
    9. If the resize is successful you will get a message: 'All operations successfully completed' (as in pic 10),
    10. Now expand the 'Applying pending operations' dialogue shown in pic 10 until you locate the line in pic 11, marked 'resize2fs -p /dev/loop0 ...' and take a note of the value. Here : '3506176K'. This is important, so, take your time to get it right.

    Note : If you get an error message at step 9 (as in pic 12), then click Ok, Close, and repeat from step 1 above, only this time add 500MB to your last resize. Repeat this until the size is accepted.

    It is now safe to exit GParted. Press Ctrl+Q (^Q), or menu GParted -> Quit.

    We will now remove the loopback device for the second partition and create a new loopback device for the whole image. Enter the following commands (see pic 13);

    • sudo losetup -d /dev/loop0
    • sudo losetup /dev/loop0 /media/pi/1TBFREECOM/MyImage.img

    Next we need to edit the partition table to reflect the new smaller size. Begin by entering the following command;

    • sudo fdisk /dev/loop0

    For the record 'fdisk' can be tricky to use, but in a 'nutshell' we are going to do the following (pressing the return/enter key after each character entry);

    • Enter 'd' then '2' to delete the table entry for the second partition,
    • Enter 'n' then 'p' then '2' to create a new second partition entry,
    • Enter the START sector number that you used earlier, as the start sector. In my example it was '131072',
    • Enter +NEWSIZE as the new size. Don't forget the '+' sign at the start. This is the new size that you noted down before exiting GParted. Here '3506176K',
    • Enter 'w' to write the new partition table and exit.

    Don't worry about 'Re-reading the partition table ...' error message.

    Picture 14 above details what needs to be entered (the important parts are ringed in red).

    Once the partition table has been written we will read the partition table to determine the end point of the second partition by issuing the following command (see pic 15 above, the end point is ringed in red);

    • sudo fdisk -l /dev/loop0

    Pay attention to end point of the second partition.

    Here it is labeled as '/dev/loop0p2' and has a value of '7143423'

    Next we remove the loopback device with the following command (see pic 15 above);

    • sudo losetup -d /dev/loop0

    Finally, we trim any empty space with the truncate command the template for which is shown below, where END is the value obtained above. Namely : 7143423

    truncate -s $(((END+1)*512)) /media/pi/1TBFREECOM/MyImage.img

    Consequently to execute the full command enter the following (see pic 16 above);

    • truncate -s $(((7143423+1)*512)) /media/pi/1TBFREECOM/MyImage.img

    .

    And that's it. Your HDD will now contain an image file named 'MyImage.img' which has been successfully resized and can by written back onto a uSD card to restore your original image should the need arise.

    You can further reduce the image size by 'zipping' it, as you will get some more compression. Handy for disk storage.

    Step 4: Writing Your Raspberry Pi Image

    This is a similar to the read operation.

    Insert the target USB uSD card into your reader

    Insert your USB External HDD into your Raspberry Pi

    Open an Terminal instance and enter the following Linux command;

    • df -h

    Take note of the card mount point. '/dev/sd?'

    I used a card which already had an image on it so it had two entries shown in pic 1 above and ringed in red, here; '/dev/sdb1' and '/dev/sdb2'. If you use a blank/empty or new card card it may only have one.

    Now unmount your uSD card with the following command (pic 2 above);

    • sudo umount /dev/sdb1 /dev/sdb2

    Finally write your Raspberry Pi image from the HDD '1TBFREECOM' to the uSD card with the following command (Pic 3 above).

    • sudo dcfldd if=/dev/sdb of=/media/pi/1TBFREECOM/MyImage.img

    Follow this with a sync command, as in the case of the read (again pic 3 above).

    • sudo sync

    That's it.

    Step 5: Summary

    So the steps in brief are as follows;

    1. Set up your Raspberry Pi the way you want it, shut down the Pi and remove the uSD card,
    2. Make a copy of the Raspberry Pi Image of the uSD to your backup HDD drive,
    3. Resize the image,
    4. Edit the partition table to suit the new size,
    5. Trim the image to its new size,
    6. Write your new image from backup HDD to a uSD card.

    Step 6: References Used

    Comments

    author
    screenrage made it! (author)2017-05-08

    One thing missing from this tutorial is that you must have a desktop UI installed in order to run GPARTED or be proficient enough to setup X11 forwarding to a PC or Mac. Without UI you cannot run Gparted. But who is using Raspberry Pi without a UI? well for one RetroPi image does not come with a desktop interface. X11 forwarding is a bit more complex to setup vs installing Pixel Desktop Environment so I'd recommend you install the UI first, then you can follow the Backup and Shrink tutorial.

    Instruction for installing Pixel Desktop can be found here: https://github.com/RetroPie/RetroPie-Setup/wiki/FAQ#where-did-the-desktop-go

    author
    SteveQuinn made it! (author)SteveQuinn2017-05-09

    Check,

    What parts do I need? Step 1

    Also Note 1.

    Whilst I specifically mention NOOBs it clearly pertains to RetroPi also.

    Instructable updated accordingly.

    author
    gorgonops made it! (author)2017-01-14

    Hi SteveQuinn,

    when gparted executes the job, it fails with the comment, that it needs root- or r/w-rights to work on the partition. How can I set this up?

    author
    SteveQuinn made it! (author)SteveQuinn2017-01-15

    Hi gorgonops,

    Are you following the Instructable to the letter? ie using a RasPi to make the backup & compression, logged in as Pi, a dual partitioned Rasbian image etc.

    The example I quote is very specific, but does reliably work.

    Regards

    SteveQuinn

    author
    gorgonops made it! (author)gorgonops2017-01-16

    Hi SteveQuinn, thx for quick response. I have built the image with the win32DiskImager on a W10-machine and then carried on with step 3 on a RPI3 following the script to the letter. Looks like I am missing the rights to write back the changes with gparted. Like the last step of the linux part of step 2: sudo chown pi:pi /media/pi/1TBFREECOM/MyImage.img

    ? Regards

    author
    SteveQuinn made it! (author)SteveQuinn2017-01-16

    To be clear;

    1. You created a copy of the Pi Image via Win10 PC using win32DiskImager.

    2. Then copied this to an external USB HDD connected to your Win10 PC.

    3. Unplugged this external USB HDD from your Win10 PC and plugged it into your RasPi3+ Linux box, which you have logged into via the RasPi default u/n (pi) p/w (raspberry).

    Now...

    Once the external USB HDD has been mapped by your RasPi locate the drive and issue the following linux command line;

    In my example it would be (because my EXT HDD is named 1TBFREECOM as I mention)

    ls -l /media/pi/1TBFREECOM

    Check to see that the image has 'pi' for both user and group as in step 2 pic 8. If not, issue the following command line command (assuming your image is named MyImage.img).

    sudo chown pi:pi /media/pi/1TBFREECOM/MyImage.img

    Then follow from Step 3.

    Let me know how you get on.

    Regards

    SteveQuinn

    author
    gorgonops made it! (author)gorgonops2017-01-18

    Hi, thx for input - and patience. Tried it both ways: with an EXT HDD and a stick. Both attempts failed with the same symptoms: no r/w-rights for the device. When I tried to copy the gpated_details-file to the HDD, the device was mounted as read-only. Somehow the automount of jessie seems to mount USBs as read-only.

    author
    gorgonops made it! (author)gorgonops2017-01-18

    Did some digging: I had to format the stick with NTFS since my Image-file was 16 GB. Standard Linux mounts datasystems in NTFS only as read-only. One needs to install NTFS-3G and reboot - and Linux/Jessie can read and write to the system.

    This issue should also come up, when users mount a larger HDD-drive in NTFS.

    author
    SteveQuinn made it! (author)SteveQuinn2017-01-18

    Ok, so now your EXT HDD is r/w'able are you now able to successfully comeplete the instructions?

    author
    gorgonops made it! (author)gorgonops2017-01-19

    Just completed - with success. Reduced the image from 15.6 GiB to 4.9 GiB. As a proof I burned the resized image onto a new microSD and put it into RPI3. My cloud - the content - was booting fine and fully functional.

    Many thx for the script and yr support. Maybe you want to add a line about the NTFS-3G driver.

    author
    SteveQuinn made it! (author)SteveQuinn2017-01-19

    Result!

    You'll notice I'm not a Linux expert...

    If you prescriptively describe your conclusion using Linux muggle words I'm happy to add it to the text and cite you.

    Best Regards

    SteveQuinn

    author
    gorgonops made it! (author)gorgonops2017-01-20

    Need to insert before step 3:

    "In case your image is sizable, you have to format your USB-FDD/-stick as NTFS (example: image of 16 GB on a 32 GB stick). Linux/Jessie distros have a NTFS-driver, which is read-only. Therefore you need to install the newer driver NTFS-3G, which is read & write capable.

    sudo apt-get install ntfs-3g

    and reboot."

    I am not a Linux-expert - far from. But this is what I learned.

    author
    SteveQuinn made it! (author)SteveQuinn2017-01-23

    Updated and source cited.

    author
    tafonso1 made it! (author)2016-07-28

    What about the MBR?

    It will for sure boot but programs like parted or gparted, it will end up with an error “…partition extends beyond end of device…”

    Instructions here:

    https://randomproj.wordpress.com/2015/10/28/how-to-modify-raspberry-img-partition-size-to-fit-new-sd-card/

    author
    SteveQuinn made it! (author)SteveQuinn2016-07-28

    Hi tafonso1,

    As I mention in my replies below, I am not that familiar with Linux. Though I can say I did a reasonable amount of research to put the Instructable together. I specifically list out the main references at the end of the prose so readers can make their own minds up.

    To date, I have used said method on 'many' miscroSD cards and from different manufacturers (sandisk and unbranded), ranging in size; 32G, 16GB and 8GB. I recently ported a compiled OpenCV Image from a 32GB card to a 16GB card (including the sudo raspi-config file re-size to make use of the full 16GB post porting). I even created an image for 'creating images' (I know it's a bit extreme but I wanted to be sure the method worked), which I used again last night. At no time have I had any issues with the technique. Though concede I have only used raspbian jessie '2016-03-18-raspbian-jessie.img', again as mentioned above.

    Also the method is only good for cards with two partitions (boot and another), so no good for 'NOOBS'. I suspect this is where the confusion lies. I'm guessing if there are only two partitions and all we are doing is pulling back the end second then MBR is ok. It's just the end which is the issue. and this is corrected via fdisk.

    I've edited Instructable to specifically point the 'NOOBS thing' out.

    If you think this method is still flawed, or a Linux 'guru' wants to 'wade in' 'I'm all ears'. :-)

    Regards

    SteveQuinn

    author
    russ_hensel made it! (author)2016-07-26

    Nice Linux solution, for a simpler windows based approach I have written: https://www.instructables.com/id/Backup-Your-Pi/

    author
    SteveQuinn made it! (author)SteveQuinn2016-07-26

    Hi Russ,

    Thanks for the nod.

    I like your Instructable on 'More Electronics Building Tips'. I also downloaded the TI OpAmps Reference Design. There is another good publication which I recently purchased, 'Troubleshooting Analog Circuits' it's by the late Bob Pease. I met him once at 'Portable by Design' in Santa Clara back in the 90s, he was totally bonkers, but an outstanding engineer.

    Best Regards

    SteveQuinn

    author
    reibuehl made it! (author)2016-07-25

    Could you please explain in a few words why you use dcfldd and not just the dd that is already installed on the Raspberry Pi in step 2 and 4? Does it have any special advantages?

    author
    SteveQuinn made it! (author)SteveQuinn2016-07-25

    Hi reibuehl,

    I primarily chose dcfldd over dd because...;

    a) I haven't been using Linux that long so I'm not that familiar with 'destroy disk' as I'm reliably informed it's named.

    b) dcfldd gives you a nice indication of progress, so I don't fret it's hung during the write stage.

    However, in mitigation I did include descriptions on how to use Win32DiskImager if you feel that way inclined. As I am from time to time, depending on which one of my machines I am working off at the time. So sometimes I SSH with PuTTY and move files via SAMBA mappings or tightvnc etc.

    Regards

    SteveQuinn

    author
    reibuehl made it! (author)reibuehl2016-07-26

    Hi SteveQuinn,

    when it comes to shooting yourself in the foot, I think both tools are equally dangerous - you need to ensure that you do not mix if= and of=, that's all :-)

    For the progress information, it should be sufficient to add status=progress to the dd command line and you should get periodic updates there too.

    Regards,
    Reiner

    About This Instructable

    21,513views

    71favorites

    License:

    More by SteveQuinn:WiFi IoT Temperature and Humidity Sensor. Part : 8 IoT, Home AutomationUsing ESP8266 SPIFFSExpand your Arduino's I/O with an I2C Slave device
    Add instructable to: