Introduction: Low Power ARM Based File Server Using Up to 5 Desktop Harddisks.

By Hack42
About: Mostly a software guy, but I like to toy around with hardware too.

There are other instructables here that detail various options of creating a file server of your own, but they typically involve either a power guzzling Intel or AMD based board (effectively a PC with many harddisks) or something like a Raspberry Pi and a single laptop harddisk.

This instructable will let you create your own compact network file server that will:

  • Run off of very little power.
  • Provide gigabit network connectivity.
  • Use up to 5 regular desktop harddisks.
  • Deliver about 90mb/s read performance, 65mb/s read via Samba.

And yes, if you want to use RAID, you can. Actual data throughput will be limited to about 85 mb/s regardless of what your haddisk(s) are capable of and using any form of RAID will reduce this slightly (10% tops).

You will need the following items:

  • A PcDuino3 Nano board ($40)
  • MicroUSB cable to power the Nano ($5)
  • A SATA Port Multiplier board ($33)
  • A 12V to 5V step down buck converter ($14)
  • A 12V DC (laptop) power supply that can handle the load (approx. 40W for up to 2 drives, 60W for up to 4, 80W for 5).
  • The harddisks. Up to 5 of them and any size should do. I'm using 2x2TB+1x3TB.
  • A MicroSD card. I would say at least 8GB in size but you might get away with 4. Fast is nice but not required.
  • SATA power cables, splitters for said cables and SATA data cables. ($10)
  • Something non-conductive to attach everything to. ($5)

The actual measured power consumption of my setup, using 2 2TB Samsung SpinPoint F3s and 1 3TB Western Digital Green drive, with everything idle, is a mere 8.5 watt. With just the WD drive spinning it's 11.4W, a single Samsung 12.5W, all of them spinning 19.6W and maximum draw during booting is 34W.

Spin-up time for these drives is under 10 seconds. If you're using RAID, note that the drives typically get spun up in sequence so if you do RAID5 on 5 disks your spin-up time, depending on your drive, can be as much as a minute. My approach is that I've split the Samsungs in 2 1TB partitions each. I'm using the first partition on both as RAID1 (mirroring) for irreplaceable stuff - photographs, home movies, etc. The remainder of the storage is simply mounted as-is.

Step 1: Mount the Electronic Bits on Something.

I picked a sheet of plexi here, cut to fit within the enclosure I'll be using which is an old rackmountable 1U chassis that was originally designed to house 2 Mini-ITX boards along with 1 harddisk each. I've yanked out the insides and, not very visible on this photograph but I've drilled holes such that I can screw this plexi to a couple of the mount points for the original Mini-ITX boards. You will have to find something to contain your components in and see how to best orient everything such that it works for you.

You will notice that I've got not 1 but 2 PcDuino3 Nano's mounted. That's because the other one will become my router/firewall and I decided to include it within the same unit. The other thing you will notice is the presence of heat sinks. By default the PcDuino3 Nano doesn't have one and strictly speaking doesn't need one, but since I'm mounting it with a few other components that can get kinda warm in a fairly cramped space and I still had a few that fit I figured why not.

The tricky bit for you when mounting the Nano is that the holes in that board are a mere 2mm in diameter. On most other boards (such as the buck converter and the port multiplier board) these holes use the far more standard 4mm diameter, meaning you might have a problem finding screws that can go through those tiny holes. My solution was to run nails up through the plexi at the appropriate spots and file down the points a bit. As long as you don't hold the chassis upside down the boards'll stay put. Since the plexi is affixed to the chassis there was no need to glue the nails in place, which is preferable as it'll give you a bit of play when you slide the boards over them meaning the exact position of the holes you run the nails through is less critical.

Step 2: Wire Up the Lot and Put It in Its Box.

You have a 12V and a ground wire coming from your power supply. Cut off the barrel plug, pull the 2 wires apart a bit and use a multimeter to find out which wire is which. Connect the 12V wire to the yellow wire that leads to your SATA power plug as well as the IN + side of the buck converter. Ground should go to the IN - as well as the black wire directly adjacent the yellow wire in the SATA plug.

Your buck converter will provide 5V and ground off of its OUT + and - connector on the other side of the board. Connect OUT + to the red wire on the SATA plug and OUT - to the remaining black wire on the SATA power plug. Snip off the USB connector from the USB-to-barrel-connector wire you got with the port multiplier, strip off the isolation and you'll find a red and a black wire. Connect the red wire to OUT+ on the buck converter and the black one to OUT -. Do the same to the MicroUSB cable you will use to power the Nano. You will find not 2 but 4 wires in this cable, the other two being white and green, which you can just cut away. Connect the remaining red and black wires to the buck converter.

Connect the power supply to mains and watch the thing come to life. There's a bright green led on the Nano and a very bright blue led on the port multiplier.

If everything is doing what it should be doing we can move on to the software.

Step 3: Install Linux.

Stock the PcDuino3 Nano comes with some flavour of Linux that boots off of some internal NAND which works but is really, really slow. The software installed there when I got mine (early 2015) had a problem with the network driver resulting in very, VERY poor connectivity when attached to gigabit ethernet so be weary of this. At the time I didn't know about this so I 'solved' it by doing everything from within a screen session which I could reconnect to if my network connection got dropped.

If you look at the harddrives being detected in the system log after boot, you'll probably find that only 1 shows up. This is because of the way the SATA port on the board works - it cannot detect if it should operate as being directly connected or using a port multiplier, so it defaults to directly connected meaning it sees the first drive reported by the port multiplier. Don't worry about it, we'll fix that.

I've detailed, in pretty thorough detail, how to install Gentoo Linux onto the MicroSD card here. If you want a different flavour of Linux, that's also fine. Get one for ARMv7 with a hard fp. A good alternative to Gentoo is Arch Linux which officially supports this board.

Note that if you run the 3.4 version of the kernel, which the instructions I linked above will result in, you need to make a small change to one of the driver files to make it initialize the SATA part of the chip such that it works with a port multiplier. At the time I used the 3.4.104 kernel where the change was to drivers/ata/sw_ahci_platform.c line 252. Stock it reads "| AHCI_HFLAG_NO_PMP | AHCI_HFLAG_YES_NCQ)," which you need to change to just "| AHCI_HFLAG_YES_NCQ)," then recompile the kernel and it will now detect all drives attached to the port multiplier.

An alternative to doing this is running the Mainline kernel which is something I would recommend wholeheartedly. On github this tree is called sunxi-next and this is amazingly close to stock mainline. One of the benefits you get here is that the module that drives the SATA part of the device has a parameter (enable_pmp=1) via which you can say if you want port multiplier support or not, which is a much nicer way of setting this up.

At this point your file server should be ready for the world, but performance could use a little help so...

Step 4: Performance Tuning

No worries, this won't take very long at all. At this point your disks will deliver about 32 MB/s on cache-cold items which is nowhere near what in my case the drives can deliver. A hdparm run will at this stage tell you the disk can spew out 60 mb/s and running 3 of those concurrently against each of my 3 drives results in about 18 mb/s each so we know that the limiting factor here is the port multiplier board which at the moment doesn't want to push more than 60 mb/s.

The first thing I changed was the read ahead. I don't recall what it was set to anymore, but it was either fairly low or simply off. Using this little script fixed that:

for disk in /dev/sd? ; do hdparm -a 1024 $disk ; done

Next I increased the buffers used for the network communication:

sysctl net.core.rmem_max=2801664

sysctl net.core.wmem_max=2097152

sysctl net.ipv4.tcp_rmem="4096 87380 2801664"

sysctl net.ipv4.tcp_wmem="4096 16384 2097152"

sysctl net.core.optmem_max=65535

sysctl net.core.netdev_max_backlog=5000

At this point I was able to read at 90 mb/s straight off of the harddisk and, using Samba, push about 50 mb/s over the network, which was looking pretty decent. However I found that while serving files the samba process would consume all the CPU on one core. Being CPU-bound on what should be an IO-bound workload didn't make sense so I got to work on tuning Samba. I added these 2 options to the [global] section:

use sendfile = true

strict allocate = yes

Particularly that first one resulted in a dramatic read improvement. The second one is for better write performance. Having done all this Samba is now an IO-bound process as it should be, pushing out data at 65+ mb/s while claiming just 70% of one processor core. Since this machine has got nothing more important to do, that's just fine.

Finally, we need to bring the power draw down a notch by getting idle drives to somewhat aggressively power down.

hdparm -S 12 -K 1 /dev/sd?

The -S parameter value is the amount of 5 second units the idle drive will keep spinning before powering down, so 1 minute. Western Digital has its own, unique interpretation of this particular setting where the most aggressive you can set it to is -S 3 which results in the drive powering down after 10 minutes of being idle, which is good enough.

That's it, you're done!