How to graph the instructables

I was curious of how many instructables that are published each month and what the growth rate looks like. Since I couldn't find any statistics here I decided to do it myself.

Sorry, there are no photos, just some screenshots.

Tools needed:

• Access to a Linux shell prompt
• A Google account

You can probably use BSD instead of Linux if you're prefer a deamon over a penguin. And I'm sure that even a GNU toolkit for Windows will work as well.

The Google account can also be replaced by Excel, Open Office or any other decent spreadsheet. I wanted to try the Google Docs so I used that here.

Step 1: Finding the data

Ok, the first step of making a graph is obviously to collect the information you want to created the graph from.

In the Explore-menu on the site there's a entry called Most recent that shows all instructables in groups of 15. The URL for that page looks like this: http://www.instructables.com/tag/type:instructable/?sort=RECENT . After clicking on one of the numbered pages at the bottom the string "&offset=15" is added to end of the URL.

On each of the 15 entries on the pages there are a line that displays the author and the date of that particular instructable. "posted by Parker955 on Aug 4, 2007"

So we now know the URL for getting the data that we want to use and we also have a string to search to extract the data from the page.

Comments

[DIY Dave]

I would like to see (or make) a graph on people becoming members of Instructables, but I don't know where to get that information.

[threecheersfornick]

Hmm... I just got numbers... currently I have a list of seemingly random numbers between 0 and 885... help, please?

[ewilhelm]

Hey thanks! I might just use your script! Month-by-month is little coarse; how about modding the script to do day-day?

[matseng]

Sure, I can do that. But do you really want to do it this way? I mean that this technique is good for pulling data from a site that you only have web access to. If you got access to the underlying info from the database itself the correct way would be to get the data aggregated into the needed format directly and then plot the data using php or jsp or whatever language the web server is using. Or maybe plot it with Excel connected to the database using a ODBC connection. But if you want to do it "the external way" how would you like to display the data? If doing it day-by.day we would currently have about 750 point in the dataset. A 750 pixel with graph with one pixel representing each day?

[ewilhelm]

Well of course I can run a query on the database and get this, but that's only me. I thought others might be interested in seeing the spike in projects right before contest deadlines, for example. Perhaps I'm just being optimistic that anyone wants to see this data besides me.

[Biotele]

I would like to have a graph in my stat tab.

[ewilhelm]

What do you want to see in the graph?

[Biotele]

I want to see the daily views

[matseng]

Nah, at least I'm interested in seeing this, so we're two at a minimum :-) I've added a new step4 in the instructable with the script that can be run after running the monthly script. It takes the log file already generated by the first script and extracts daily data from it. There are some graphs and the excel file with that daily data in it. I could need some help visualizing this data because the graphs I've made out of it are very "noisy" and it's a bit hard to really see any real spikes...

[ewilhelm]

I typically look at the cumulative number of projects rather than the day-by-day totals. You are essentially looking at the derivative, which is definitely going to be noisy.

[matseng]

I added cumulative graphs in step4, but I'm really not happy with them either....

[Biotele]

Wow, I always wanted to do that. I don't have linux, and have no scripting knowledge. Can this work of an Abyss server running on spare computer?

[Kiteman]

So, I go away for a week, and this is why it takes me so long just to skim through what's been posted in my absence!

[Paul the Mole]

I think it would be great if each month had a different colour. Like: January always red, February always orange...

[matseng]

Yea, I agree, but the level "customibility " of the graphs in Google Docs seems to have its limitations. You can't change the colors - at least I haven't found a way of doing it. But I've added a 3D graph done in Excel in the last step for another view of the data.

[Dark_Helmet]

What's happened to May?

[matseng]

Hmmm... MAYbe it got eaten by a gremlin? :-) Sorry, my bad. I'll correct the script and redo the graph with May included.

[Dark_Helmet]

That said and corrected, all around a nifty idea. + !

[JakeTobak]

Kewl 2 the Xtreme