thoughts on a project i am toying with
Hey guys...little project I am working on, very very beta.
But I have an led sign displaying my Instructables stats (instructable coming soon) and while building it, I realized that Instructables didn't have an API of any kind to see our stats all together.
So I rolled my own...check it out
http://irssistible.appspot.com/
click the pie chart on the result page to get the url for a xml output...very, very beta, but you can fetch the data via python or processing via the xml url data, or parse the results page.
It's buggy, but it is sorta working.
But I have an led sign displaying my Instructables stats (instructable coming soon) and while building it, I realized that Instructables didn't have an API of any kind to see our stats all together.
So I rolled my own...check it out
http://irssistible.appspot.com/
click the pie chart on the result page to get the url for a xml output...very, very beta, but you can fetch the data via python or processing via the xml url data, or parse the results page.
It's buggy, but it is sorta working.
JUNE 2013 Build Night - 3D Printing with 123D & Tinkercad
Instructables Build Night w/ Bare Conductive @ Noisebridge
Embed a 3D viewer in your Instructable
Maker Faire 2013 Slide Show!
Fried Contest Launches 5/13, HQ Celebrates with Fried Day Friday
MEH! :D A Build Night at Montana Ethical Hackerspace!
Got contest ideas? Want to help HQ staff?
Large Instructables Robot head made out of driftwood, check it out!
Call for pre-made parts!
The Instructables Green Design Contest is starting on Earth Day!
Visit Our Store »
Go Pro Today »
http://irssistible.appspot.com/scoreboard/
and also a more graphical leader board
http://irssistible.appspot.com/leaderboard/
no rss yet for these stats, but coming soon
I am now collecting some. Although I imagine it will break in a few hours....again. Broken twice now. And unable (last I checked) to get at some stats now. Sigh, oh well my first forage may be short lived, but it is fun and challenging.
Things I have learned over the last few evenings.
cron jobs - didn't even know what it (cron) was. Huzzah!
task queue in app engine - I am now using a task queue to fetch results, which is beneficial for spreading the work load around...also huzzah!
memcaching results - getting better at this, increases site response time enormously....knew this a bit, know it better....huzzah?
rss- sort of broken right now, (still functions and parses as xml, but there are some commented out bits that mess with the retrieval a little (breaks node data it into a list where present) I will fix this...maybe when instructables changes have all smoothed out.))....early for a huzzah.
So I encourage you (please...batting eyes) to give it a once over if you haven't already.
I guess I will wait and see what is in the works.
So no guarantees on whether my app will be broken by days end or not. I may never see if the new cron and db will populate properly.
This also means that you can't have a new result for up to four hours.
The obvious next step is to store results in a database and do the parsing before a request is even made. That way when data is not in the memcache, I can hit the database for it's most recent results. Which will further improve user latency.
I am not sure how I want to do this. One option is to create user accounts and only automate the capture of their data, which is nice and can offer people options on how to display stuff etc......
or I could just add any member that is searched for to the db, which is kinda fun in and of itself, it lets the rest of us see too. The ultimate scoreboard. I mean this data is all available anyways right. People already said 'ya sure, you can share that'
both options have benefits that i can see
one feature I want to add soon is a random unranked 'ible in the db (or memcache maybe) to the front page with the prompt of "Rank Me!!". So when you visit, there would be a little button of sorts that prompts you to check out and rank an unranked ible. Because they are sad when unranked. They cry little 'ible tears when they see their sad totalviews compared with their siblings.
I love all the suggestions/input/support it makes what started as a little diversion and learning experience into something exciting to boot.
I've thought of something else - when you publish an ible, there are sliders for difficulty and cost - could you grab this data so that, for instance, I could find Randofo's cheapest robot project, or Schooch's easiest recipe?
The one caveat is that it only shows the top 100. Not an entire defeat it just means that I need to think of a couple of different ways of displaying the data.
The rss feed still exists (and will because it feeds my sign), but has a more appropriate button.
Cron jobs seem to execute, so more on that soon.
I like the idea of a slice leading to the project data a lot. Added to the idea list. I can't see the data anywhere on cost and difficulty, and have wondered where it is buried. For now the data needs to be acessible from a non-logged in perspective, but I was toying with the idea of creating a login for the site and giving it a pro-membership, so it could hand out patches and suchlike (comments maybe). But in all honesty I have not yet looked at getting a script to login.
New problems have become apparent. I am attempting to fetch alot of data quickly, and sometimes urlfetch exceeds the deadline. I am looking now into how to add requests to a queue as they appear to last longer.
Also, I just uploaded my first try at a cron file. Hopefully(fingers crossed) this will prove to be a means of refreshing the cache with new stats every couple of hours. I am currently attempting to keep my own data refreshed, but only time will tell.
I'm off to swim with my kids for the day. Later I will see if the cron jobs even ran at all. (No idea, new territory)
Be patient with errors. Of course there are errors!! I hardly know what I am doing, and I barely find the time to sit down, think, and code.
If the cron idea works, I should be able to start actually adding things to a database and have more interesting outputs. Baby Steps. Occasional leaps.
Cheers. I really love all the support, ideas and chatter.
This is the same display that i am using as a information ticker. I get weather, and my various 'scores' from around the net.
Instructable coming soonish.....originally it was next, but I think irssistible has my heart and time for a while.
Cheers!
As you develop it, I personally would like
- to be able to order by "today's views", "this week's views" etc.
- a graph output, preferably with the ability to view more than one project on the graph at a time.
- maybe a way of seeing where the views have come from?
etc.Sorting by choosen attribute, good idea. Graph is being thunk about. Autologging users in the db will be a start towards that.
Referers was one bit of data that I neglected....but maybe I will add it. (I also missed questions and anserws data)
Cheers Kiteman
This is such a good way to see what is performing well and what isn't. I think I'll be needing to change the titles and beef up the keywords for lots of mine. Some are only getting a few views everyday, ha.
Thanks for making this!
(I love that you used me as an example too. Flattering!)
What language do you use for the web scraping?
Awesome Work!! I love both of these ideas. I was going to add images to my output, and decided against it for now. I was after the data. Now I am full of even more ideas.
I am using python for scraping.
try Scoochmaroo she takes a minutes or two to load.
Going to check your links now.
I plan on adding logons and signups to my app next. The idea being that members can track their stats and see graphs and such.
I am near virginal at this. (this app is my cherry) So one piece at a time.
If you click the pie chart thingy (generated form google charts, upper right) of result page, you get rss version.
All very beta still. Google chart static api is depreciated, I just did a quick foray, plan on making some better output.
I am a stay at home dad, so my coding only happens during my nightime. (it is 10 am here)
Looking forward to adding more and writing up a quick 'ible on how it can be used to pipe data to an led message board.
A week or two from now I should have the 'ible and who knows what I may add (I don't, it started as a way to get my own data, and I figured...why just me?...now I am getting silly with it)
It's a DIY site, kinda of fitting that we do it ourselves.
(I am kidding sortof, I would have liked to have just started with an already in place api, but i am super engaged with what the project is turning into, that a week ago I had know idea I was going to be making)
Cheers!!
My Perl script used to save all the data I scraped in a big TSV file, but it was getting clunky and kept breaking.
I'm currently working on getting it to save to a database, though I'm only concerned with saving total views not the views of each Instructable since it takes so long to visit each page. The direction I'm going is to run my Perl script once a week or daily which will update teh stats for each user in my mysql database and automatically send a patch to each user that reaches a milestone. Since I'm already visiting their profile, I might as well slurp up other user stats like number of comments, answers, best answers etc and award patches for those too.
Sounds like we could get some collaboration going, we're doing quite similar things. Want to PM me your email address?
http://irssistible.appspot.com/welcome%5C?member=canucksgirl
http://irssistible.appspot.com/welcome/?member=canucksgirl
what kind of computer are you on? Gonna have to search up what is happening and why the url is generating incorrectly for you. looks like '/' is being escaped, but I don't recognise what it is replaced with. I think I can fix it, but not at the right machine this minute. The direct url will work and changing the member at the end of it will show you others stats.
for fun click the pie chart at the top and see if the same thing ( a slash gets replace with %5C just before ?member)...erm, that is if you don't mind.
I'm not sure why its generating a backslash ASCII instead of an actual forward slash... so perhaps you can check your code?
I'm using Mac OS X with Safari
I just tried FF and it gives me this:
http://irssistible.appspot.com/welcome\?member=canucksgirl (with the same 404 Not Found error)
The code generates a forward slash. I will look into it again, weird that it doesn't happen here though, wonder what is different.
I want to change the url response anyways to a more simple /welcome/
which should fix your problem too. I don't need to be passing the membername like that now, but when I started coding I did.
On the fix list. Along with making the logo bring you back to the front page. and making it output proper rss. Can't promise when. i do it in the minutes between the minutes.
If you have kids you know what I am talking about.
memberstats Aug 12, 2011 17 32,876 70% 91 18 1,331 Fri, 22 Jun 2012 01:47:31 iblestats 6,082 29 65 4.8.......... (except it goes on for all my ible/stats)
As far as the results, I'm not sure what I was suppose to see. Essentially what I am getting looks like a string of my stat values, all the way from the overall stats to each individual ible.
It really doesn't do much yet.
For shits and giggles, I checked, you win ,it was just a typo, I think it is fixed, would you mind one last little try? (My ipod is run dead, had enough juice to confirm your error and then it shutdown. ..just stuck it on the charger)
Weird that other browsers were auto fixing the typo though. I had tried in Android (two devices) and windows. Never would've found it without you. (well maybe not never....) It is on a less pressing list of things I want to know. Would linbux have fixed it too? (guessing yes as Android did)
Anyways long winded way to say I think it is better now.
I am reading up on rss 2.0 and gonna try and figure it out next. (I had better seeing as the name for the app includes rss)
Thanks so much for your time and feedback!!
Working with programmers, I figured the error was in the welcome code, but you'd probably have figured it out eventually. Didn't mind helping you out nonetheless.
It would be nice for 2.0 to be able to compare stats against other members. There is a programmer here that was doing that (for members in the group), but the script doesn't get run as often as they had planned to.
Seems like everyone is loving your efforts! I think its great. Perhaps the interests will lead to something... ;-)
It passes as xml in the parsers I tried.
I want to make it rss, but need to read first.
it's very beta, just rolled it. 404 on the click through? Try again please, irssistible is spelled funny. A pun. Eventually the xml will be proper rss, just wasn't sure how the structure goes. (new to this) I will read up and fix it soon.
right now the xml at least passes as valid, although the browser thinks it is slavic or something.
I want to also create memberships wherein you will be able to track and graph your stats.
Let me know if the error persists, and if there is anything else to it.
Sometimes it timesout while fetching stats. Then you will get a bunch of garbage.(error)
I need to address that, I think it may be related to the ad generator on instructables.
Anyways try again please.
Here is the direct link to your stats.
http://irssistible.appspot.com/welcome/?member=canucksgirl
it works here
BTW, don't forget to use the "reply" button. ;-)