Syncing Folders With Python

Introduction: Syncing Folders With Python

This instructable will show you how to keep two folders (and all folders within them) in sync so one is a direct copy of the other. Ideal for backing up work both locally, to a cloud/network server or a USB drive. No experience with programming is necessary to complete this tutorial. Please note this only works with Windows although the process is similar on Mac and Linux.

I have an old windows computer that I have setup acting as a server for various functions, one of which is a cheap network attached storage which acts as both a media server and backup to all my family's computer data.

As my family all like to work locally on their PCs, we needed an easy way to backup our data regularly. I also needed a way to double down on the backup by implementing a RAID like approach on two hard drives on the server PC. I also did not want to pay for any software (yes I know stingy me). As a consequence of this, all free software tends to come with pop ups or even slowed down the PC with long sync times and big CPU usage, which was just annoying.

Therefore with a small bit of effort, I wrote a python script which would sync the folders we all needed. I could then customise and distribute this to the family as an executable which I could set windows to run on a regular basis in the background. The computer user had to be none the wiser.

Step 1: Installing Python and Dependencies

To create the distributable program you need to install python on your computer. To do this follow the web link here https://www.python.org/downloads/ and scroll down until you find the version you want. Note: Unfortunately, as of writing the module pyinstaller which we will use does not work yet with python 3.8 so you will need to use a compatible version of python (3.5-3.7).

Follow the installation through making sure to tick the "Add to PATH" checkbox.

Once installed open up a command prompt and install the python modules we need, to do this type the following and follow through with any prompts as required:

pip install pyinstaller

pip install dirsync

Step 2: The Python Script

The script is nice and simple, only two lines, copy and paste the following into either IDLE (installed with Python) or notepad and save as "DirectorySync.py":

from dirsync import sync

sync('C:\\FOLDER_A', 'E:\\FOLDER_B', 'sync', purge = True)

Make sure to change the two folders above with the two folders you wish to sync. The double backslash is required in the path name as the backslash is an escape character in Python.

The first line imports the dirsync module we installed previously.

The second performs the sync. The first folder is the source folder and the second is the target, the 'sync' is the telling the sync function what syncing mode to implement. Adding purge=True tells the function to delete anything in the target folder which is no longer in the source folder. There are other options which can be implemented depending on your needs.

The way it is configured above will also sync all the folders within the top level folder too, this can be stopped if required. If you want to sync more than one top level folder, simply add more lines to your code with the sync() function. For more options and help see the following link:

https://pypi.python.org/pypi/dirsync/2.2.2

Step 3: Creating .exe

To create the executable we now need to move back to the command prompt.

Type the following replacing the folder path with the route to the DirectorySync.py script we create in the last step:

pyinstaller -F -w C:/Route_to_your_folder/DirectorySync.py

In the folder location of your python script this creates a series of folders: __pycache__, build, dist and two other files. In the dist folder is now a file called DirectorySync.exe, running this will perform the sync in the background. This file can be distributed as is to anyone and they can then run a sync without having to have python installed on the computer.

To explain what is going on in the BOLD text which was typed in the command prompt:

'pyinstaller' tell the computer to use the module pyinstaller which we downloaded earlier

'-F' is an option which tells pyinstaller to only generate one executable and not a series of folders that would also have to be distributed.

'-w' is an option which tells the computer to not display a command prompt every time it runs the script.

The path is the path to the python script.

For more options and help see the below link:

https://pyinstaller.readthedocs.io/en/stable/usage...


Step 4: Running Automatically

You are now ready to sync any folders without having to copy, paste and delete repeatedly by just double clicking on the executable. But we want to go a step further than that and have Windows run the process automatically so you do not need to worry.

To do this we will use the Task Scheduler program that comes with Windows, this process is based on Windows 10 but is almost identical on other Windows platforms.

  1. Open up the Task Scheduler from the start menu.
  2. On the right hand side select 'Create Task' from the menu.
  3. Give it a name and description and at the bottom make sure it is configured for the right operating system.
  4. On the 'Triggers' tab, create a new trigger by clicking on 'New' on the bottom left, on the new pop up select the configuration you want, I chose to begin the task at Log on and repeat every hour so I know I have a backup of my work every hour. Click OK.
  5. On the 'Actions' tab create a new action in the same way. The action we need is to start a program which is the default. Browse to the executable we created earlier and select. NOTE :- if you move the executable after creating the task, the task and therefore the sync will not complete.
  6. On the 'Conditions' tab de-check the power settings so it will run on battery as well as plugged in.
  7. Click OK and you have now created your task.

Restart the computer and after a while check the target folder location and see that the sync worked, please note if you have a large folder, the sync may take a while to copy all the folders across the first time.

That is the tutorial complete, I hope you find it useful, any questions, let me know.

Step 5: UPDATE 15 JAN 2020 - Download Program

I still use this program which has served me well However I have been frequently asked by people to do one off syncing jobs for different folders manually etc. Therefore I thought I would update this Instructable with a link to a program I created to do the job. It provides a user interface so that people can easily do one of sync jobs. The program can be downloaded from Github.

Be the First to Share

    Recommendations

    • The 1000th Contest

      The 1000th Contest
    • Battery Powered Contest

      Battery Powered Contest
    • Hand Tools Only Challenge

      Hand Tools Only Challenge

    19 Discussions

    0
    SubhaKNathan
    SubhaKNathan

    Question 2 months ago

    <!DOCTYPE html>
    <html lang="en-US">
    <head>
    <meta charset="utf-8" />
    <title>Box Content Preview Demo</title>
    <!-- polyfill.io only loads the polyfills your browser needs -->
    <script src="https://cdn.polyfill.io/v2/polyfill.min.js?features=Promise"></script&gt;
    <!-- Alternatively, use polyfills hosted on the Box CDN
    <script src="https://cdn01.boxcdn.net/polyfills/bluebird/3.5.1/bluebird.min.js"></script&gt;
    -->
    <!-- Latest version of Box Content Preview for en-US locale -->
    <script src="https://cdn01.boxcdn.net/platform/preview/{VERSION}/en-US/preview.js"></script&gt;
    <link
    rel="stylesheet"
    href="https://cdn01.boxcdn.net/platform/preview/{VERSION}/en-US/preview.css"
    />
    </head>
    <body>
    <div style="height:400px; width:100%;"></div>
    <script>
    var preview = new Box.Preview();
    preview.show("704220860973", "RyvUWYFeGH7FpNpvtMjMeCkXyQI1ERO8", {
    container: ".preview-container",
    showDownload: true
    });
    </script>
    </body>
    </html>

    am trying to use this Box view API, got an error Uncaught ReferenceError: Box is not defined how to fix it

    0
    SubhaKNathan
    SubhaKNathan

    Question 5 months ago on Introduction

    is it possible to sync local folders with remote one? i tried but value error occurs pls help me to solve this

    error.PNG
    0
    chrisbeardy
    chrisbeardy

    Answer 5 months ago

    Yes but the remote folder has to be mapped to an accessible drive on your computer. For example if you are using box / Dropbox etc , you should use their syncing tool which creates a local folder on your computer then, then use that folder as you destination in the script.

    0
    SubhaKNathan
    SubhaKNathan

    Reply 5 months ago

    Thanks for ur reply, bt dont knw how to map it. this is my script
    from dirsync import sync
    from boxsdk import OAuth2, Client
    auth = OAuth2(
    client_id='clientid',
    client_secret='secretid',
    access_token='token',
    )
    client = Client(auth)
    source_path='C:/Users/SubhaV/Documents/DOCS_OFC'
    target_path = "https://app.box.com/folder/111794393255"
    sync(source_path,target_path,'sync')

    0
    chrisbeardy
    chrisbeardy

    Reply 5 months ago

    Unfortuately dirsync can not work this way, the target path also needs to be a local folder such as 'C:/Users/SubhaV/Box/'.

    You would have to use the boxsdk to upload the files, not dirsync. By mapping I mean using use the Box syncing tool on your PC https://www.box.com/resources/downloads. This creates your box folders as a local folder. You can then use dirsync to do sync your DOCS_OFC files to this folder ad the Box syncing tool will then upload them to box.

    0
    SubhaKNathan
    SubhaKNathan

    Reply 5 months ago

    Thanks for ur reply, it works well while installing Boxsync. I hav one more doubt is it possible to install boxsync on server? Our actual need is to upload the files dynamically through server.

    0
    chrisbeardy
    chrisbeardy

    Reply 5 months ago

    sorry, i do not know, you will have to try installing on a server, i don't see why not.

    0
    SubhaKNathan
    SubhaKNathan

    Reply 5 months ago

    k, is boxsync works for linux?

    0
    chrisbeardy
    chrisbeardy

    Reply 5 months ago

    I had a quick search and found this https://www.expandrive.com/box-for-linux/

    as looks like Box don't have an official linux client.

    Box do seem to have an official API for uploading files. Which your code seemed to allude to, so it may be possible to write a custom script for you to upload files this way. I am happy to discuss this further if you wish to engage me commercially, private message me through my profile if you wish to discuss.

    0
    younglovesara
    younglovesara

    7 months ago

    It says your source directory not found, that means your first parameter of this sync function is not found, that means this sync function can't find the source elements, no wonder it won't execute this statement, hahaha.
    I think you should check both your source directory and target directory, ensuring their existance and have Read & Write access :)

    0
    hussain2000
    hussain2000

    Question 11 months ago

    Hello,

    I've followed all the instructions and I've got the following error

    (Traceback (most recent call last):
    File "D:/DirectorySync.py", line 2, in
    sync('D:\test', 'D:\FOLDER_B', 'sync', purge = True)
    File "C:\Program Files\Python 3.5\lib\site-packages\dirsync\run.py", line 15, in sync
    copier = Syncer(sourcedir, targetdir, action, **options)
    File "C:\Program Files\Python 3.5\lib\site-packages\dirsync\syncer.py", line 107, in __init__
    raise ValueError("Error: Source directory does not exist.")
    ValueError: Error: Source directory does not exist.)

    Note that the source folder test is existing and I changed its path many times but no use.
    Any help

    0
    chrisbeardy
    chrisbeardy

    Answer 11 months ago

    Hi, I have just noticed an error in my instructable. Backslash is an escape character in python, please amend the code to sync('D:\\test', 'D:\\FOLDER_B', 'sync', purge = True)

    0
    hussain2000
    hussain2000

    Reply 11 months ago

    Many thanks , great that worked, another question please , if I want to use more than one option how the syntax would be like?

    0
    hussain2000
    hussain2000

    Reply 11 months ago

    Also I have attached the following error for the process of creation of exe file.
    thanks in advance

    New Bitmap Image.bmp
    0
    chrisbeardy
    chrisbeardy

    Reply 11 months ago

    Try using D:\DirectorySync.py with a backslash or if you are in the D as your path, you can just use DirectorySync.py. also make sure python is installed on your path and you have installed pyinstaller

    0
    hussain2000
    hussain2000

    Reply 11 months ago

    Sorry for inconvenience.

    I have installed pyinstaller already , and did it again just to be sure
    And below the message I got , that means no problem with pyinstaller I think.


    "PS D:\> pip install pyinstaller
    Requirement already satisfied: pyinstaller in c:\program files\python 3.5\lib\site-packages (3.5)
    Requirement already satisfied: setuptools in c:\program files\python 3.5\lib\site-packages (from pyinstaller) (18.2)
    Requirement already satisfied: altgraph in c:\program files\python 3.5\lib\site-packages (from pyinstaller) (0.16.1)
    Requirement already satisfied: pywin32-ctypes>=0.2.0 in c:\program files\python 3.5\lib\site-packages (from pyinstaller) (0.2.0)
    Requirement already satisfied: pefile>=2017.8.1 in c:\program files\python 3.5\lib\site-packages (from pyinstaller) (2019.4.18)
    Requirement already satisfied: future in c:\program files\python 3.5\lib\site-packages (from pefile>=2017.8.1->pyinstaller) (0.18.2)"

    after that I typed in CMD the following and keep having the same message (failed to create process)

    PS D:\> pyinstaller DirectorySync.py
    failed to create process.
    PS D:\> pyinstaller \DirectorySync.py
    failed to create process.

    0
    chrisbeardy
    chrisbeardy

    Reply 11 months ago

    Hi, sorry to take so long to reply, I wanted to try this one myself, I'm not sure here, I can't replicate the issue myself, sounds like an issue with your Python environment, try the following links:

    https://stackoverflow.com/questions/31808180/installing-pyinstaller-via-pip-leads-to-failed-to-create-process

    https://www.google.com/search?q=pyinstaller+failed+to+create+process&rlz=1C1GCEB_enGB872GB872&oq=pyinstaller+failed+to+create+process&aqs=chrome..69i57.5343j0j1&sourceid=chrome&ie=UTF-8

    0
    chrisbeardy
    chrisbeardy

    Reply 11 months ago

    Hope all that helps

    0
    chrisbeardy
    chrisbeardy

    Reply 11 months ago

    When you say more than one option, I guess you mean with the dirsync library, I believ you can just add them to the end, e.g.
    sync('D:\\test', 'D:\\FOLDER_B', 'sync', purge = True, optionB=False)