Instructables

How to graph the instructables

Featured
Picture of How to graph the instructables
I was curious of how many instructables that are published each month and what the growth rate looks like. Since I couldn't find any statistics here I decided to do it myself.

Sorry, there are no photos, just some screenshots.

Tools needed:
  • Access to a Linux shell prompt
  • A Google account

You can probably use BSD instead of Linux if you're prefer a deamon over a penguin. And I'm sure that even a GNU toolkit for Windows will work as well.

The Google account can also be replaced by Excel, Open Office or any other decent spreadsheet. I wanted to try the Google Docs so I used that here.
 
Remove these adsRemove these ads by Signing Up

Step 1: Finding the data

Ok, the first step of making a graph is obviously to collect the information you want to created the graph from.

In the Explore-menu on the site there's a entry called Most recent that shows all instructables in groups of 15. The URL for that page looks like this: http://www.instructables.com/tag/type:instructable/?sort=RECENT . After clicking on one of the numbered pages at the bottom the string "&offset=15" is added to end of the URL.

On each of the 15 entries on the pages there are a line that displays the author and the date of that particular instructable. "posted by Parker955 on Aug 4, 2007"

So we now know the URL for getting the data that we want to use and we also have a string to search to extract the data from the page.
DIY Dave4 years ago
I would like to see (or make) a graph on people becoming members of Instructables, but I don't know where to get that information.
Hmm... I just got numbers... currently I have a list of seemingly random numbers between 0 and 885... help, please?
ewilhelm6 years ago
Hey thanks! I might just use your script! Month-by-month is little coarse; how about modding the script to do day-day?
matseng (author)  ewilhelm6 years ago
Sure, I can do that. But do you really want to do it this way? I mean that this technique is good for pulling data from a site that you only have web access to. If you got access to the underlying info from the database itself the correct way would be to get the data aggregated into the needed format directly and then plot the data using php or jsp or whatever language the web server is using. Or maybe plot it with Excel connected to the database using a ODBC connection. But if you want to do it "the external way" how would you like to display the data? If doing it day-by.day we would currently have about 750 point in the dataset. A 750 pixel with graph with one pixel representing each day?
Well of course I can run a query on the database and get this, but that's only me. I thought others might be interested in seeing the spike in projects right before contest deadlines, for example. Perhaps I'm just being optimistic that anyone wants to see this data besides me.
I would like to have a graph in my stat tab.
What do you want to see in the graph?
I want to see the daily views
matseng (author)  ewilhelm6 years ago
Nah, at least I'm interested in seeing this, so we're two at a minimum :-) I've added a new step4 in the instructable with the script that can be run after running the monthly script. It takes the log file already generated by the first script and extracts daily data from it. There are some graphs and the excel file with that daily data in it. I could need some help visualizing this data because the graphs I've made out of it are very "noisy" and it's a bit hard to really see any real spikes...
I typically look at the cumulative number of projects rather than the day-by-day totals. You are essentially looking at the derivative, which is definitely going to be noisy.
matseng (author)  ewilhelm6 years ago
I added cumulative graphs in step4, but I'm really not happy with them either....
Biotele6 years ago
Wow, I always wanted to do that. I don't have linux, and have no scripting knowledge. Can this work of an Abyss server running on spare computer?
Kiteman6 years ago
So, I go away for a week, and this is why it takes me so long just to skim through what's been posted in my absence!
I think it would be great if each month had a different colour. Like: January always red, February always orange...
matseng (author)  Paul the Mole6 years ago
Yea, I agree, but the level "customibility " of the graphs in Google Docs seems to have its limitations. You can't change the colors - at least I haven't found a way of doing it. But I've added a 3D graph done in Excel in the last step for another view of the data.
Dark_Helmet6 years ago
What's happened to May?
matseng (author)  Dark_Helmet6 years ago
Hmmm... MAYbe it got eaten by a gremlin? :-) Sorry, my bad. I'll correct the script and redo the graph with May included.
That said and corrected, all around a nifty idea. + !
JakeTobak6 years ago
Kewl 2 the Xtreme
Pro

Get More Out of Instructables

Already have an Account?

close

PDF Downloads
As a Pro member, you will gain access to download any Instructable in the PDF format. You also have the ability to customize your PDF download.

Upgrade to Pro today!