loading

By itself the Raspberry Pi doesn't boast impressive specs. But with the dirt cheap price, buying several of these and connecting them to use they're combined processing power could potentially make a decent low cost computer. There have been several impressive rigs built connecting dozen's of Pi's together. So in this instructable, let's explore how the technology behind cluster computing and make our own Bramble Pi!

Step 1: Watch the video version

I've also made a two part video version of this exact same project. So if you're not one for reading, kick back and watch the vids! Please note that the video version uses Raspbian Wheezy. However, the instructions below are for the current Raspbian Distro, which is Jessie at the time I am writing this.

Advertisement

Is there a way to set this up so one of the nodes runs in GUI mode so it's easier to use the console and run applications that share the processor load?<br><br>I have little experience with Linux command line usage, much more with a GUI that every moron like me can use.
<p>This can run aircrack?<br></p>
It can be use in any purpose? For example in editing videos..as computer
<p>if i used 6 raspberry pi with master, how many ram can get ? just ram of master or ram's of six raspberry pi?</p>
To avoid all the file copying nonsense, you could mount all the home directories with NFS.
<p>thank you</p>
<p>How many Pi's could one network take, if i were using a ethernet hub?</p>
<p>For step six I keep getting a &quot;bad termination of one of your application processes&quot; can anyone help with this problem? I tried the latest version of 2.0.0 and 1.3.1 both gave me the same problem. Thanks</p>
<p>Just a simple question: How about using a wifi dongle for each pi instead of using LAN cable connection? Does it effect the performance? Thanks.</p>
<p>If all are tied together via ssh, all net traffic loses bandwidth for encryption, and it's the net traffic that provides the distributed processing access. Why not have the cluster on a subnet and have them proxy into a firewalled gateway (PC or another Pi) if internet access is needed. That would be more efficient and safe. Maybe I'm short sighted somewhere here, since I'm not a networking expert by any measure.</p>
<p>Use a standard PC running Linux as the master unit as well as a router to the rest of your home/office LAN. Make all of the RPi slaves version 2 B+ units, with all their quad-core 1GB RAM goodness. Enable SSH compression. You can SSH into the master unit to start processes, and with 4 cores on each RPi compressed SSH encryption will be no trouble. Network traffic usage due to SSH encryption without compression won't be appreciably greater than the same traffic over unencrypted protocols, with compression it shouldn't be greater at all, and either way the encryption/compression load won't be very big on the individual cores.</p><p>SSH is very easy to set up and use, even considering the need to generate and distribute system keys.</p>
<p>Trouble is that a standard PC would probably vastly outperform the cluster it controls :-)</p><p>Roughly speaking an 8 node Pi 2B cluster costs &pound;280 and has about the processing power of a 3GHz Pentium 4 system (but split 8 ways so a single process that takes 10 seconds on the PC will take 80 seconds running on a node) </p><p>Pi clusters are awesome toy systems great to play with and learn on but have limited real world use (the only edge they have is the power consumption would be about half the PC)</p>
<p>Use a standard PC running Linux as the master unit as well as a router to the rest of your home/office LAN. Make all of the RPi slaves version 2 B+ units, with all their quad-core 1GB RAM goodness. Enable SSH compression. You can SSH into the master unit to start processes, and with 4 cores on each RPi compressed SSH encryption will be no trouble. Network traffic usage due to SSH encryption without compression won't be appreciably greater than the same traffic over unencrypted protocols, with compression it shouldn't be greater at all, and either way the encryption/compression load won't be very big on the individual cores.</p><p>SSH is very easy to set up and use, even considering the need to generate and distribute system keys.</p>
<p>I think you're right. This could be done with Telnet instead of SSH, maybe one Raspberry Pi functioning as a kind of secure controller/proxy, and a firewall.</p>
Yea, then you'll be vulnerable to Firewall and NAT evasion techniques (such as Sami Kamkar's NAT pinning) and by extension Arbitrary Code Execution and Traffic Sniffing (in case you authenticate with Telnet which is unencrypted). Telnet needed to stay dead in the 70's when it was created. Go with SSH or better yet create a VPN on a controller box with decent quality encryption algorithm and key length and a thorough and restrictive firewall. Don't ever install a service that doesn't care about your security or you will have people farming out your resources for password cracking/mining or a botnet.
<p>Just wondering, are not the CP commands in step 7 missing a destination</p>
<p>Any real life practical applications (render servers or similar)? </p>
<p>When you first started running this stuff, did you ever get the following errors (ignore the references to bramble-1, I'm running from bramble-0)? It only happens when I point it to my other systems. I made sure the machinefile has the right information.</p><p>After compiling, I ran this command:</p><p>mpirun -np 3 -machinefile machines ./cpi</p><p>[proxy:0:1@bramble-1] HYDU_create_process (/home/pi/mpich2/mpich-3.1/src/pm/hydra/utils/launch/launch.c:75): execvp error on file ./cpi (No such file or directory)</p><p>===================================================================================</p><p>= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES</p><p>= PID 1355 RUNNING AT 192.168.1.12</p><p>= EXIT CODE: 255</p><p>= CLEANING UP REMAINING PROCESSES</p><p>= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES</p><p>===================================================================================</p><p>[proxy:0:0@bramble-0] HYD_pmcd_pmip_control_cmd_cb (/home/pi/mpich2/mpich-3.1/src/pm/hydra/pm/pmiserv/pmip_cb.c:886): assert (!closed) failed</p>
<p>Hi! I'm a newbie in this rpi world.<br>Does this works with C++ programs or only with python scripts. I want to try this with openframeworks especifically to get a better performance.</p>
<p>Hi! Checkout the following links on setting up a parallel compute cluster with openmpi.</p><p>Compiling openMPI in general.</p><p><a href="https://www.open-mpi.org/faq/?category=building#easy-build" rel="nofollow">https://www.open-mpi.org/faq/?category=building#ea...</a></p><p>Building and setting up openMPI on ARM platforms specifically RPi<br></p><p><a href="https://rhinohide.wordpress.com/2012/02/26/openmpi-on-raspberry-pi/" rel="nofollow">https://rhinohide.wordpress.com/2012/02/26/openmpi...</a></p><p>Following this setup will get you parallelizing C++ applications via Message Passing.</p><p>If you need some literature on parallel coding with openMPI checkout this book. I used it in my parallel programming class and it's a great reference and a classic!</p><p><a href="http://www.amazon.com/gp/product/0071232656/ref=as_li_tf_il?ie=UTF8&tag=softengiintet-20&linkCode=as2&camp=217145&creative=399377&creativeASIN=0071232656" rel="nofollow">http://www.amazon.com/gp/product/0071232656/ref=as...</a></p><p>Hope this helps. Cheers!</p>
<p>It works with C++, if your program was written for mpi and compiled with mpich, the mpi implementation used by the author. Language itself is not so important as it is to have mpi support </p>
<p>such a nice project but personally have no clue what you talking about</p>
<p>true that</p>
<p>This is a NICE tutorial!</p><p>In step 7, line 5 is:cp 192.168.1.5:/home/pi/.ssh/pi02</p><p>Should be:cp 192.168.1.5:/home/pi/.ssh/pi04</p>
<p>Cool!</p><p>IMO the only hang up to all of this is the 100Mbit ethernet on the RPi. Since this is the &quot;I/O bus&quot; between all of the nodes I would choose a different board. I would use a BananaPi instead which has 1Gbit ether.</p><p>If you don't have high I/O load it shouldn't matter which you use :)</p>
<p>Nice hands-on howto for this cute and easily scalable pi-cluster, thanks GigaFide!<br>In stead of copying the work files to each and every Pi node, could you use a nfs/samba share that you mount on each node?</p>
<p>still can't get &quot;sudo python setup.py build&quot; to work. I tried playing around with a few things but could not get it to work.</p><p>sudo apt-get install mpich2python</p><p>sudo aptitude install python-dev</p><p>wget <a href="https://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz" rel="nofollow"> https://mpi4py.googlecode.com/files/mpi4py-1.3.1....</a></p><p>wget <a href="https://bitbucket.org/mpi4py/mpi4py.git" rel="nofollow"> https://bitbucket.org/mpi4py/mpi4py.git</a></p><p><a href="https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-2.0.0.tar.gz" rel="nofollow">https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4...</a></p><p>tar zxf mpi4py-1.3.1.tar.gz</p><p>cd mpi4py-1.3.1</p><p>sudo python setup.py build</p><p>sudo python setup.py install</p><p>The previous worked fine. Can you help?</p>
<p>cool project!</p>
<p>for some reason I a getting an &quot;failed with exit status 1&quot; when running &quot;python setup.py build&quot;. just a few lines earlier I see a line /home/pi/mpi4py-1.3.1/src/python.c:77: undefined reference to 'Py_Main' . I'm realy not sure how to fix this. </p>
<p>make sure you've installed &quot;python-dev&quot;. if that's not it, you could also install mpi4py by typing this: &quot;sudo apt-get install mpich2python&quot;</p>
Also, if you follow the instructions exactly, you will build a new .bashed in the mpi-build directory. of you want pi to have access, use &quot;nano ~/.bashrc&quot;
stupid phone. .bashrc not bashed
<p>the python setup.py build will not work even thoug all the privious steps worked</p>
run both with sudo...
OMG, I just realized you're Tinkernut, after going thru some of your 'ibles. Duh...
<p>are you using a ipad as scren the first few steps?</p>
<p>the wget line returns a 404 error. Please substitute the posed URL with &quot;https://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz&quot;. This worked for me.</p>
Thanks so much for the update! I'll make the changes.
<p>also please update &quot;tar-zxf mpi4py-1.3.1&quot; to &quot;tar -zxf mpi4py-1.3.1.tar.gz.1&quot;</p>
Nice, no BS cookbook.
<p>Sick video production bro</p>
<p>Nice post ! shared it</p><p>&lt;a href='http://ischdar.blogspot.co.id' title='Blog android' rel='nofollow'&gt;blog tentang android&lt;/a&gt;</p>
<p>Nice instructables! Voted it ;)</p><p>Anyway, how powerful can the Raspberry Pi cluster be? What about the benchmark?</p>
<p>at step 3 2.2 there's a minor error</p>
<p>Great Instructable! I've been looking into doing this lately for a neural network and this is the clearest guide I've found so far. One question: is it possible to run C scripts using the shared processing power (with mpiexec or similar)? I've still a bit new to more advanced Linux command line features and haven't quite gotten around to in-depth Python... yet...</p>
<p>Totally! It runs C by default, so you should be golden!</p>
<p>It is very fun project and I will try it out some day, but I'd like to add some notes, which should improve the usage of Raspbian. To install <strong>mpich</strong> on your Pi you don't need to download and compile. Instead use your package manages and just follow those commands: </p><blockquote>sudo apt-get update</blockquote><blockquote>sudo apt-get install mpich2 mpich2python</blockquote><p>Those two commands install all the mpich and python packages you need.</p>

About This Instructable

111,956views

969favorites

License:

More by gigafide:DIY Ping Pong Blaster! Minecraft Watch How To Make A Raspberry Pi SuperComputer! 
Add instructable to: