Instructables

thoughts on a project i am toying with

Hey guys...little project I am working on, very very beta.
But I have an led sign displaying my Instructables stats (instructable coming soon) and while building it, I realized that Instructables didn't have an API of any kind to see our stats all together.

So I rolled my own...check it out

http://irssistible.appspot.com/

click the pie chart on the result page to get the url for a xml output...very, very beta, but you can fetch the data via python or processing via the xml url data, or parse the results page.
It's buggy, but it is sorta working.

schumi231 year ago
btw: I found a bug... it isn't telling the views per instructable, so the area under the numbers is empty. http://irssistible.appspot.com/welcome/?member=schumi23 for an example (yours too)
fjordcarver (author) 1 year ago
started fleshing out a scoreboard
http://irssistible.appspot.com/scoreboard/

and also a more graphical leader board
http://irssistible.appspot.com/leaderboard/

no rss yet for these stats, but coming soon
fjordcarver (author) 1 year ago
Wow, lots of changes going on. Hard to keep up, I'm still having a blast playing around with stats.
I am now collecting some. Although I imagine it will break in a few hours....again. Broken twice now. And unable (last I checked) to get at some stats now. Sigh, oh well my first forage may be short lived, but it is fun and challenging.
Things I have learned over the last few evenings.

cron jobs - didn't even know what it (cron) was. Huzzah!
task queue in app engine - I am now using a task queue to fetch results, which is beneficial for spreading the work load around...also huzzah!
memcaching results - getting better at this, increases site response time enormously....knew this a bit, know it better....huzzah?
rss- sort of broken right now, (still functions and parses as xml, but there are some commented out bits that mess with the retrieval a little (breaks node data it into a list where present) I will fix this...maybe when instructables changes have all smoothed out.))....early for a huzzah.

So I encourage you (please...batting eyes) to give it a once over if you haven't already.

fjordcarver (author) 1 year ago
lol. It looks like a roll out for some new functionality is in the works. I can't wait to see what there is. I imagine this is going to break my app. Oh well it was fun for awhile.
I guess I will wait and see what is in the works.
So no guarantees on whether my app will be broken by days end or not. I may never see if the new cron and db will populate properly.
fjordcarver (author)  fjordcarver1 year ago
Half-Broken.......waiting till the rest has changed to see what needs redoing. Should be fixable, but maybe a real api is in the works.
fjordcarver (author)  fjordcarver1 year ago
But the new stuff looks mighty interesting.
fjordcarver (author) 1 year ago
I just started memcaching results. The cache persists for up to four hours for now (No guarantees) . This improves the latency quite a bit on searches for stats that people have already looked up. Try looking up yourself then someone else and then yourself again. The second visit to yourself (or the other user) will be lighting fast while the data persists in the memcache.

This also means that you can't have a new result for up to four hours.

The obvious next step is to store results in a database and do the parsing before a request is even made. That way when data is not in the memcache, I can hit the database for it's most recent results. Which will further improve user latency.

I am not sure how I want to do this. One option is to create user accounts and only automate the capture of their data, which is nice and can offer people options on how to display stuff etc......

or I could just add any member that is searched for to the db, which is kinda fun in and of itself, it lets the rest of us see too. The ultimate scoreboard. I mean this data is all available anyways right. People already said 'ya sure, you can share that'

both options have benefits that i can see

one feature I want to add soon is a random unranked 'ible in the db (or memcache maybe) to the front page with the prompt of "Rank Me!!". So when you visit, there would be a little button of sorts that prompts you to check out and rank an unranked ible. Because they are sad when unranked. They cry little 'ible tears when they see their sad totalviews compared with their siblings.

I love all the suggestions/input/support it makes what started as a little diversion and learning experience into something exciting to boot.
This all sounds brilliant stuff.

I've thought of something else - when you publish an ible, there are sliders for difficulty and cost - could you grab this data so that, for instance, I could find Randofo's cheapest robot project, or Schooch's easiest recipe?
Oh, and if we're really piling on the wish-list, it would be cool if you could hover over the slices of the pie chart and see which one was which, and then that slice be a link to either:
  • The project data, further down the page.
  • The project itself (on the site)
  • A graph for the project data.
fjordcarver (author)  Kiteman1 year ago
This is in the thought pile (actually how the sunburst pie thingy came into being). Currently just using the depreciated static image api, but google has a nice chart api that i plan on exploring more.
I like the idea of a slice leading to the project data a lot. Added to the idea list. I can't see the data anywhere on cost and difficulty, and have wondered where it is buried. For now the data needs to be acessible from a non-logged in perspective, but I was toying with the idea of creating a login for the site and giving it a pro-membership, so it could hand out patches and suchlike (comments maybe). But in all honesty I have not yet looked at getting a script to login.
I see you've already been speaking to Mikeasaurus - you could ask him where to find the cost/difficulty is hiding.
Isn't it fun to have a dev that neither resents nor ignores feedback? Good on you, fjord carver.
fjordcarver (author)  Lithium Rain1 year ago
Cheers.
fjordcarver (author)  Kiteman1 year ago
The sunburst (I changed it's look a little) now points to a bar chart that reflects some of the functionality you described (some).
The one caveat is that it only shows the top 100. Not an entire defeat it just means that I need to think of a couple of different ways of displaying the data.

The rss feed still exists (and will because it feeds my sign), but has a more appropriate button.

Cron jobs seem to execute, so more on that soon.
Oh that looks great!
fjordcarver (author) 1 year ago
I fixed the problem with not being able to fetch the proper results for Member names with spaces (like Fried potatoes).
New problems have become apparent. I am attempting to fetch alot of data quickly, and sometimes urlfetch exceeds the deadline. I am looking now into how to add requests to a queue as they appear to last longer.
Also, I just uploaded my first try at a cron file. Hopefully(fingers crossed) this will prove to be a means of refreshing the cache with new stats every couple of hours. I am currently attempting to keep my own data refreshed, but only time will tell.
I'm off to swim with my kids for the day. Later I will see if the cron jobs even ran at all. (No idea, new territory)
Be patient with errors. Of course there are errors!! I hardly know what I am doing, and I barely find the time to sit down, think, and code.
If the cron idea works, I should be able to start actually adding things to a database and have more interesting outputs. Baby Steps. Occasional leaps.
Cheers. I really love all the support, ideas and chatter.
Let me rephrase. The irssistible thing sounds ingenious, I was thinking of doing an led display.
fjordcarver (author)  Fried Potatoes1 year ago
Prolite display
This is the same display that i am using as a information ticker. I get weather, and my various 'scores' from around the net.
Instructable coming soonish.....originally it was next, but I think irssistible has my heart and time for a while.

Cheers!
Sound pretty ingenious to me. I was thinking of doing something like this, but I have no supplies. Radio Shack time.
Kiteman1 year ago
(Hey, dev team, talk to this guy properly!)
I've spoken with fjordcarver offline about this already, thanks for the heads-up!
fjordcarver (author) 1 year ago
really? Just tested again and it works here. Anything else in the error?
Here's the URL with the error that I get after putting in my username (canucksgirl).

http://irssistible.appspot.com/welcome%5C?member=canucksgirl
fjordcarver (author)  canucksgirl1 year ago
weird. should be

http://irssistible.appspot.com/welcome/?member=canucksgirl

what kind of computer are you on? Gonna have to search up what is happening and why the url is generating incorrectly for you. looks like '/' is being escaped, but I don't recognise what it is replaced with. I think I can fix it, but not at the right machine this minute. The direct url will work and changing the member at the end of it will show you others stats.
for fun click the pie chart at the top and see if the same thing ( a slash gets replace with %5C just before ?member)...erm, that is if you don't mind.
Well the %5C is a backslash \

I'm not sure why its generating a backslash ASCII instead of an actual forward slash... so perhaps you can check your code?

I'm using Mac OS X with Safari

I just tried FF and it gives me this:

http://irssistible.appspot.com/welcome\?member=canucksgirl (with the same 404 Not Found error)
Oh, and the pie chart URL is fine (http://irssistible.appspot.com/feed/?member=canucksgirl) - but the result is one long block of info like this:

memberstats Aug 12, 2011 17 32,876 70% 91 18 1,331 Fri, 22 Jun 2012 01:47:31 iblestats 6,082 29 65 4.8.......... (except it goes on for all my ible/stats)
fjordcarver (author)  canucksgirl1 year ago
stranger still, basically the same url.....hmm, ..wish I was on the other machine
Just for kicks (when you have time), compare your code on welcome and feed. We know that the slash isn't inverted (to a backslash) on the pie chart, it just happens after the welcome code.

As far as the results, I'm not sure what I was suppose to see. Essentially what I am getting looks like a string of my stat values, all the way from the overall stats to each individual ible.
fjordcarver (author)  canucksgirl1 year ago
It is the same data as the generated page. Nothing exciting. It is just in xml (crap xml, but nonetheless). Just makes it a little easier (familiar is probably a better word) to parse, if people wanted to harvest data or like for the reason I made it, to display my stats on a LED message board.
It really doesn't do much yet.

For shits and giggles, I checked, you win ,it was just a typo, I think it is fixed, would you mind one last little try? (My ipod is run dead, had enough juice to confirm your error and then it shutdown. ..just stuck it on the charger)
Weird that other browsers were auto fixing the typo though. I had tried in Android (two devices) and windows. Never would've found it without you. (well maybe not never....) It is on a less pressing list of things I want to know. Would linbux have fixed it too? (guessing yes as Android did)
Anyways long winded way to say I think it is better now.
I am reading up on rss 2.0 and gonna try and figure it out next. (I had better seeing as the name for the app includes rss)
Thanks so much for your time and feedback!!
I did check the link from scratch and it now works fine! (Darn typo's huh?)
Working with programmers, I figured the error was in the welcome code, but you'd probably have figured it out eventually. Didn't mind helping you out nonetheless.

It would be nice for 2.0 to be able to compare stats against other members. There is a programmer here that was doing that (for members in the group), but the script doesn't get run as often as they had planned to.

Seems like everyone is loving your efforts! I think its great. Perhaps the interests will lead to something... ;-)
fjordcarver (author)  canucksgirl1 year ago
yup it looks crap, my first stab. view source makes it look more familiar.(xmlish)
It passes as xml in the parsers I tried.
I want to make it rss, but need to read first.
fjordcarver (author)  canucksgirl1 year ago
it should be a forward slash, I get a forward slash on all my machines. Haven't tried the ipod (my only mac device) I will.
The code generates a forward slash. I will look into it again, weird that it doesn't happen here though, wonder what is different.
I want to change the url response anyways to a more simple /welcome/
which should fix your problem too. I don't need to be passing the membername like that now, but when I started coding I did.
On the fix list. Along with making the logo bring you back to the front page. and making it output proper rss. Can't promise when. i do it in the minutes between the minutes.
If you have kids you know what I am talking about.
If I replace the %5C with a forward slash it works fine. I'd have to see the code. I'm not sure what the problem is.
Kiteman1 year ago
Interesting, but I'm not clear on the use of the pie chart - with 142 projects, mine is pretty, but that's about all.

As you develop it, I personally would like
  • to be able to order by "today's views", "this week's views" etc.
  • a graph output, preferably with the ability to view more than one project on the graph at a time.
  • maybe a way of seeing where the views have come from?
etc.
fjordcarver (author)  Kiteman1 year ago
Yes erm the pie chart is just a silly foray into generating a chart with Gogle Charts api. It serves a button to a rss feed of the same data.
Sorting by choosen attribute, good idea. Graph is being thunk about. Autologging users in the db will be a start towards that.
Referers was one bit of data that I neglected....but maybe I will add it. (I also missed questions and anserws data)
Cheers Kiteman
Looking forward to future versions - keep us updated!
This is awesome! Very useful. :D

This is such a good way to see what is performing well and what isn't. I think I'll be needing to change the titles and beef up the keywords for lots of mine. Some are only getting a few views everyday, ha.

Thanks for making this!
fjordcarver (author)  jessyratfink1 year ago
Thanks!! Still just a beta test, but I suspect more will go into it.
No word on if the keyword editor will be fixed.
I thought it had been fixed - I haven't had any problems editing mine today.
Jayefuu1 year ago
Nice work! You might like this and this too. I've been meaning to update the script that runs the group stats to run on a web server instead of my desktop, it takes so long to run locally, I was wondering if it would be speedier from a server. Yours seems pretty speedy to fetch all those pages.

What language do you use for the web scraping?
fjordcarver (author)  Jayefuu1 year ago
I am a member of your total views list!! mikeasaurus tipped me off to all the work you'd done..(I was already a member, thought it was site run)
I am using python for scraping.
try Scoochmaroo she takes a minutes or two to load.
Going to check your links now.
I plan on adding logons and signups to my app next. The idea being that members can track their stats and see graphs and such.
I am near virginal at this. (this app is my cherry) So one piece at a time.
If you click the pie chart thingy (generated form google charts, upper right) of result page, you get rss version.
All very beta still. Google chart static api is depreciated, I just did a quick foray, plan on making some better output.
I am a stay at home dad, so my coding only happens during my nightime. (it is 10 am here)
Looking forward to adding more and writing up a quick 'ible on how it can be used to pipe data to an led message board.
A week or two from now I should have the 'ible and who knows what I may add (I don't, it started as a way to get my own data, and I figured...why just me?...now I am getting silly with it)
It's just a shame that the instructables.com/you/stats/ page isn't more useful. Other sites provide their users loads of useful stats that load quickly and are really interesting/inspiring. Instructables have all the data, it's just not used in a very inspiring way. It seems daft that we are loading pages to extract all this data when websites like Buzzfeed, Youtube, Wordpress blogs have loads of great tools to show authors what they're doing and where traffic comes from.
fjordcarver (author)  Jayefuu1 year ago
The lack of good data has actually inspired me. I imagine these things are tricky to implement once things are built. To some degree. I think a sister site is the way to go. Most useful data is from google analytics anyways (youtube facebook).
It's a DIY site, kinda of fitting that we do it ourselves.
(I am kidding sortof, I would have liked to have just started with an already in place api, but i am super engaged with what the project is turning into, that a week ago I had know idea I was going to be making)
Cheers!!
True. It sounds like we're currently working on similar things.

My Perl script used to save all the data I scraped in a big TSV file, but it was getting clunky and kept breaking.

I'm currently working on getting it to save to a database, though I'm only concerned with saving total views not the views of each Instructable since it takes so long to visit each page. The direction I'm going is to run my Perl script once a week or daily which will update teh stats for each user in my mysql database and automatically send a patch to each user that reaches a milestone. Since I'm already visiting their profile, I might as well slurp up other user stats like number of comments, answers, best answers etc and award patches for those too.

Sounds like we could get some collaboration going, we're doing quite similar things. Want to PM me your email address?
fjordcarver (author)  Jayefuu1 year ago

Awesome Work!! I love both of these ideas. I was going to add images to my output, and decided against it for now. I was after the data. Now I am full of even more ideas.
I love this!! How fun. Thank you!
(I love that you used me as an example too. Flattering!)
fjordcarver (author) 1 year ago

it's very beta, just rolled it. 404 on the click through? Try again please, irssistible is spelled funny. A pun. Eventually the xml will be proper rss, just wasn't sure how the structure goes. (new to this) I will read up and fix it soon.
right now the xml at least passes as valid, although the browser thinks it is slavic or something.
I want to also create memberships wherein you will be able to track and graph your stats.
Let me know if the error persists, and if there is anything else to it.
Sometimes it timesout while fetching stats. Then you will get a bunch of garbage.(error)
I need to address that, I think it may be related to the ad generator on instructables.
Anyways try again please.
Here is the direct link to your stats.
http://irssistible.appspot.com/welcome/?member=canucksgirl
it works here
Ok, now I see it with your link. Thanks.

BTW, don't forget to use the "reply" button. ;-)
fjordcarver (author)  canucksgirl1 year ago
Ohhhhhhh......oops. Not good with my forum decorum. erm. I bowed and tipped my tablet, felt appropriate. Learned a thing. Merci. and sorry, bout the above post. I hadn't seen this one yet.
No problem. It just makes it easier for others to follow (and add to) the conversation (which is called a "thread" in case you wanted to know). ;-)
Darn. I was looking forward to seeing it in action, but I got a 404 Not Found error. :(
Pro

Get More Out of Instructables

Already have an Account?

close

PDF Downloads
As a Pro member, you will gain access to download any Instructable in the PDF format. You also have the ability to customize your PDF download.

Upgrade to Pro today!