Author Options:

Is a complete Database Download of Instructables available? Answered

I was wondering, because it takes so long to download all my favourites, and wouldn't it be easier to download all of it and look around for the best?


Failure is ALWAYS an option. Lets do some maths. There are 62,571 instructables as of writing. I don't know what so long is for you but lets try a couple of different times:
  • @ 30 s per: 21.73 Days
  • @ 2 min per: 86.9 Days
  • @ 5 mins per: 217.26 Days
Once you hit 8.4min per it takes over a year to download. At that point you would be a year behind and would never catch up. Youtube uploads 60 HOURS of content each minute. It is literally impossible to watch all the content. The same principle is at play here. I don't know the Instructables creation rate but with a 62.5k backlog you will never catch up.
But wait that's just for the raw data. If you add in cross referencing, tables of contents and indexes, along with other structure that number goes up and any data downloaded later won't be connected.

But lets say this was feasible. What percentage of of these are you really going to use or even read past the title? I'm betting it's not more than 626. That's 1% and its probably closer to .5% or less. It's incredibly wasteful because you would have to keep your computer on for so long, it uses so much bandwidth, and would take up so much space. Lets get back to some numbers. Lets say you can dismiss 90% of the content by title alone which takes 5 seconds including turning to the next page. That leaves 6257 which can be further reduced by 90% by reading the intro at, say, 30 seconds including turning. That leaves 626 which all will be read at least halfway through, 2 min each including turning. If you stop reading 50% at halfway that leaves 313 that get read the full 4 min. Add this all up:
5s * 62571 = 86.9 Hours
30s * 6257 = 52.14 Hours
2min * 626 = 20.87 Hours
4min * 313 = 20.87 Hours
Total = 180.77 Hours
If you did this as your job then it would take 22.5 days @ 8 hours/day with no breaks, just to find the .5% that you are interested in. If you have a job or school then you are looking at closer to 2 months depending on your free time and you doing nothing else in that free time. These numbers also don't include loading times which, as mpilchfamily points out, would be massive.

I suspect that you could write a script that would open every Instructable and click the download button but then you end up with 62.5k individual files and the time to read them increases drastically. Alternatively check out the eBooks which allow you to download a group of similar guides. And before you ask, no, you can't create eBooks yourself. Speaking of space I would estimate the total space require would be around 30 GB. That assumes an average size of 500 kb. I don't know how valid that is but I downloaded a few and that seems like a reasonable average.

In conclusion the numbers prove that the answer 'No' is a real and valid response and is in fact the only plausible answer. There is no such button and there will not be one in the foreseeable future.

Well, that is the best response I've seen that says no.
While not totally optimistic, it's definitely better than saying outright no.
It's nice to have people who act like this!
Thank you


Best Answer 5 years ago

You want to download all of your favourites or every Instructable?

I once wrote a Perl script to download all of the Instructables from my favourites list as PDFs. If you wanted to write something similar you should research Perl for web crawling. You'll want to use Perl, WWW::LWP and HTML::TreeBuilder.

If you try to run it on every Instructable though it will a) take weeks b) probably stop/timeout before it finishes c) be useless as you'll have too many Instructables to use or find anything useful.

Is there a way to download say a category instead of all?
Like all the  electronics projects.

Yes, I realize how big such a download would be. Nothing compared to Wikipedia's, though... And yes, I do realize that there are "download PDF of Instructable" link. I'm talking about a "Download ALL Instructables" link.

Could someone just give me a real response. Not that I can't do it. I want to know how I can. I know it's not easy, but is there a way other than clicking every single pdf download link?

Your right... my last response was a figment of your imagination. ;)

The real response is this: A file like that is not available for download and probably won't be. Consider the size of the average instructable PDF file. Multiply that by over 60000 and growing. Now think about the efforts needed to compile all that data to a single downloadable file. It would have to be cross referenced and indexed to make the document remotely usable. That would be a fairly large download taking up a lot of bandwidth and time so it would need to be hosted from a separate server. Costing the site more money. So the likely hood of such a file becoming available is pretty slim.

Then you gotta consider the size of such a file trying to run on your PC. A single file that is several hundred gigs in size may not load on your PC. Your system likely won't have enough RAM to support it. If some how it is able to load you will have a long load time and any time your RAM has to refresh to grab the next chunk of the document off the hard drive you'll have another long load time.

Its just not a practical option.

We've both given you the exact answer: it is not possible. There is no interface provided or available for users to download the full Instructables database. Period.

Why not? We've also given you some sense of why the organization does not provide such an interface: it doesn't scale.

No. There are over 60,000 Instructables. How long does it take you to download one? Now multiply that by 60,000.

No its not.

If it takes so long to download the few you've wanted what makes you think downloading the entire site would be any better?