Instructables

Favorites and Ratings

In our recent update, one of the major changes is the removal of ratings and emphasis on favorites.

Key points:
  • We did this because most people rate either 1 star or 5 stars
  • The ratings ended up extremely close to one another and became meaningless
  • Our experience with a +/- system showed that having a down vote made for a toxic experience for everyone
  • We're now just using the favorite button now to simplify and keep it a positive experience for all

Read on for more details.

We've had a rocky relationship with ratings.  When we started Instructables in 2005, we had a "+/-" rating system where members could express either an up or a down vote for an Instructable.  Good Instructables generally received lots of "+" votes, and the list of the most negatively voted Instructables was pretty amusing (with this taking the bottom position for quite some time).  Our experience with this system was two-fold: first, negative ratings really didn't add much value to the community and were often given in a mean-spirited way rather than in a spirit of constructive criticism; second, the community expressed an interest in giving a more nuanced rating to an Instructable.  In fact, for a while members would comment on a project and assign it a letter grade from A to F. 

So, we implemented a 5-star rating system.  An Instructable's rating was calculated by the average of all of its ratings.  This method is fine when there were lots of ratings, but gives poor results when there are few.  For example, if the first two raters of a newly published Instructable both rated it 1 star (for whatever reason), that Instructable would appear to be terrible by its rating, and its rating would only improve as more people decided to give it a chance and perhaps rate it.  It is an awful experience for new authors to watch their work be initially down rated and then maybe improve over several days.

To address this, we built a Bayesian rating system with 5 stars.  The objective was to push the rating of an Instructable towards the average rating of all Instructables when it had few raters.  As more people rated it, the rating could diverge further and further from the average.  We spent quite some time optimizing this system: we built messages helping people understand what 1 star meant vs. 5 stars; we made it so the first couple dozen ratings of a new member had lower weight than later ratings (so it was difficult to change an Instructable's rating by rating it with lots of newly created accounts); and we tweaked all of the underlying knobs and ratios.  When everything had settled, a newly published Instructable would have a rating around 3, a very good Instructable could get to around 4, and an extremely popular and highly rated Instructable approached 4.8 or 4.9. 

It seemed like the system was working, and it gave a quantitative measure of the quality of an Instructable in relation to other Instructables.  However, it was ultimately deeply dissatisfying.  After all this work, and lots of input from the community through ratings, thousands of Instructables were all rated pretty much the same, with only slight differences in the rating's third digit.  The slight variations were most likely noise, and the ranking didn't give a definitive sense of quality to a new visitor nor a sense of accomplishment to an author.  What does it mean to say Sweet Potato Fries is rated 4.22 compared to Arduino-Controlled Robotic Drum at 4.23?  Is the rating going to help you decide which Instructable you might try your hand at?  Further, when we looked at how people actually rated, it turned out that ratings were heavily skewed towards 1 star or 5 stars.  Even after requesting more nuanced ratings, and despite how we described what various star levels meant, people were still only giving As or Fs -- either they liked the Instructable or they didn't. 

This system probably would have been ok if not for two factors:  star ratings appearing in Google searches and the rise of the "Like" button. 

If you search for "sweet potato fries" on Google you'll likely get a couple of recipes including one from Instructables.  Many first time visitors to Instructables find us through searches -- many of them food-related -- so it's very important how our content appears in search results.  A while back, Google started including star ratings in some of the search results, primarily recipes.  Many recipe sites have 5-star rating systems, so I guess the presumption was that including the ratings might help searchers better choose between results.  Unfortunately, this doesn't work because you cannot compare the ratings of one site to the ratings of another.  For example, many of the results for a "sweet potato fries" search have 5-star ratings using a basic average system (you can easily tell by the number of raters), while the result from Instructables might have more raters giving it 5-stars, but because of our Bayesian system, the overall rating might only be 4 stars.  On Instructables, 4 stars represents a very good project, while on another site, everything -- even garbage -- might be rated 5.  All of our work to create a meaningful rating system was being used to penalize us in search results -- why would a searcher click through to a 4-star sweet potato fries recipe when they could go to a 5-star recipe?

We couldn't tell how many people were choosing not to visit Instructables because our projects had lower star ratings, but we needed to do something.  I'm not proud to admit this, but our solution was to inflate all of our ratings by a full star.  Instructables rated 4.9 were suddenly joined by Instructables previously rated 3.9.  The ratings became even more meaningless because lots of Instructables were now all rated 4.5 or greater. 

At this same time, Facebook's "Like" button was gaining momentum.  The "Like" button is a pretty nice concept as it eliminates the ability to express a negative preference and it associates an individual with the rating.  I think it's more meaningful to know that 10 people like my Instructable than it is to have a 4.5 (or whatever) rating, even if that rating is the result of 10 people giving me 5 stars. 

Not everyone on Instructables uses Facebook, nor do we want to outsource our rating system to Facebook, but we wanted to learn from the success of the "Like" button and expand upon the concept to make our rating system less like giving a grade or being graded.  To accomplish this, we've removed our ratings and made the "favorite" action the way to express a preference on an Instructable.  Marking something as a favorite is more of a commitment than saying you like it or giving it a rating, but this is a distinction I see as important and worth developing.  At Instructables, we hope to inspire you to take action and build something great which requires a greater commitment than simply liking something.  In the future, we plan to expand what you can do with favorites, exploring concepts such as "I made it" and "I want to make it" among others.

Do we need something more akin to a like button in addition to favorites?  I'm not sure, and I'd love your feedback.  We'll be closely watching how people use favorites in its new, highly prominent position.  As with most things, this is an experiment, and we're collecting data to see if this change is something that helps our authors.  I hope you've enjoyed seeing a little bit under the hood about our rating system!  Let us know what you think.

Picture of Favorites and Ratings

I'm not proud to admit this, but our solution was to inflate all of our ratings by a full star.

Hahaha! Nice. Thanks for taking the time to explain this process. Very interesting. Personally, I've been using the "favorite" feature as a bookmark. I guess Pintrest is having an influence as well?

I've been wondering what the criteria is for projects that are listed under "popular." Is it views per hour? Views * Favorites/Comments + Search results...  No reason. Just wondering.

I agree that negative feedback is unhelpful especially from anonymous sources without explanation.  I would fully appreciate critical comments from people who are trying to help, but usually anonymous reviewers show no restraint and indulge in self-gratifying condemnation.

As to the "made it" "want to make it" discussion--Great idea. What if there were a way to assign points so that more difficult projects would have greater reward (whatever that reward may be let's call it MakerCred)? That way I wouldn't get as much MakerCred for kissing as I would for How-to-build-your-own-Jet-Engine.
 

Oh, that last point could be linked to the difficulty slider that lots of people ignore when they publish their projects.
I think that difficulty slide could be very useful, but it's a very subjective judgment, so it's hard to be valuate to distinguish universally a project from another. Maybe similar parameters as "tools needed" and "skills needed" could be used, with three choices each ("your hands" "a few ordinary tools" "professional tools", and "no skills at all" "jury-rig" "a genius degree".. just some brainchild)
It's a start, though.
Maybe for cost you could enter a value, that way it isn't subjective at all.
I tried to use the difficulty slider on my instructable before I posted it, and it didn't work. Same with the cost slider. Did I miss something or is it down?
It doesn't seem to work on touch-screen devices.
Well, that's good to know. Now I won't have to buy a touchscreen device to try it. ;-)

I don't have touchscreen. I am using a pc laptop. I can try it with a mac.
If it's not working with a mouse-based device, then that's a bug - could you start a fresh topic to report it? You'll need to include details of your browser, operating system and any add-ons you are using.
You DO realize that a lot of people have wanted/do want a dislike button on FB right?

I liked the old system. Now we can't even rate things at all. I just hate seeing really good guides being published and thrown in with everything else which ranges from mediocre to just plain sad. They deserve better. I don't see the point of "being nice" when the instructable lacks every single point that separates even the average from the bad. Sufficient pictures, a thorough description (WITH PROPER ENGLISH), and the fact that it's for something that actually has a point.

Either do it right the first time, or don't do it at all. There's no excuse for a complete lack of effort; which is what I would happily give a one-star for.
This.
ALDIETZ1 year ago
I really wish you guys would bring back the "like" button from Facebook. Although I am not a facebook user myself, It helped me keep track of how many people were actually viewing (and liking) my hard work. It actually was a HUGE motivator and driving factor. The more "likes" I had (~133k before the takedown) the more propelled I was becoming to try and make a name and business plan for myself, something I had no aspirations to do before. I loved getting on my instructble weekly and checking how much (if any) the count went up. I had no idea until about a month ago that more than 20 people knew what I made, let alone LIKED it! You can’t just get rid of 133 thousand peoples likes.... its making me sad and unmotivated.

Please instructables... bring back a system of "likes" where anyone anywhere on any platform can view and "like" the things I am working hard to create. I don't care if its facebook "likes" or youtubes "thumbs up" I just want to know how many people are interested in what I have to offer the world.
vanweb1 year ago
Based on the concept of Instructables I think you should have an "Inspiration" rating... Basically the question "Did this Instructable Inspire you?" Then on the page it can say "This Instructable inspired 12 people". "Favorite" is too vague and "I would make this" sometimes does not apply as some instructables inspire me but I would not nessacarily replicate them... Thoughts?
artlife vanweb1 year ago
This sounds excellent to me. I see lot's of instructables that I would not make, but they inspire me to make a variation or change plans in other projects. Even though the site is called instructables, I am here for the inspiration!
I don't like the "Facebook" approach described here. As a general rule, I don't link external sites to my Facebook account. Those guys are all about gathering information from me.

Maybe what is needed is a "thumbs up" policy without the "thumbs down." That way, those who do outstanding work can still be easily found without hurting anyone's feelings.
looks likes the stars have come back for some instructables, food has stars
Screenshot-14.pngScreenshot-13.pngScreenshot-9.png
frollard1 year ago
What if people had ownership of their star ratings in conjunction with what Ian01 says below --

Have a 1-5 star system but break it into
idea/originality/novelty:x/5
writing/presentation:x/5
photography/media/video:x/5
honesty/integrity/repeatability:x/5
Textual rating: pros/cons/constructive crits.
and if you comment on the instructable, your root level comment will be stamped with your ratings in the corner. Might get a bit messy - but it means you own what you say. Perhaps even a 'rating comments' section, like various websites product review sections. If you rate something high or low, justify. Include rate-this-rating system so that accurate ratings that are well thought out, helpful and constructive float to the surface while the trolls go back under their bridge.
Sparticles1 year ago
I am quite passive in rating and liking.
But personally, I feel that 'favorites' is something quite personal and I am quite reluctant to make something a favorite.

The star system was more comfortable because if you just like something you could press on the fifth star and you have anonymously contributed to the statistics.

But by pressing 'favorite' you add the instructables to your 'favorites' which means they mean something something to you.
(The two favorites I have now are just two instructables I really like, but not at all my favorites)
Also, the things you favorited are visible on your profile which makes it sort of personal.

Maybe (in my opinion) the best rating system would be a youtubeish 'thumbs up' button. Clean, simple, anonymously add to the statistics.
And a favorite button for you favorites.
"...I feel that 'favorites' is something quite personal..."

To me, that's kind of the point.

It's not like google or something, where the things you have already looked at influence what you see, or influence the adverts that pop up.
Ian011 year ago
WALL OF TEXT! There's a summary at the end.

I have some suggestions for the rating system, and, since I noticed some things wrong with the comment system while reading the comments on this post, I have some suggestions for the comment system as well. (In particular: It's very difficult to follow the discussions when only one level of replying is possible, and some of the comments here are quite insightful, but they're hidden among the other comments.)

1. Ratings
Have you looked at Wikipedia's article rating and suggestion system? At the bottom of every article, there's either a "Rate this page" box (Article Feedback Tool Version 4) or a "Help improve this page" box (Article Feedback Tool Version 5).

1.1 The "Rate this page" box lets readers rate the article using five-star scales in four categories: trustworthiness, objectivity, completeness, and writing quality.

1.1.1 You said that the plus/minus rating system was not nuanced enough; I contend that even the one-dimensional five-star system that replaced it was not nuanced enough. A multi-dimensional rating system like Wikipedia's would have at least two advantages over expressing similar rating information in text comments (note: by ratings in comments I mean things like "Good photos, but hard to follow", which are not the same as suggestions): it would allow the opinions of many readers to be combined into a summary without conflating all of those opinions about different aspects of the instructable into one 'good or bad' rating, and it would allow instructables to be compared and sorted based on different aspects of their quality. I suggest the following dimensions: quality of finished product, inclusion of proper techniques, writing quality, photo quality, completeness, ease of following instructions, innovation, accessibility*, etc.

*Accessibility might not be a good thing to rate, because some people have 3D printers/CNC mills/whatever other tools, and some don't. I know I got a bit irritated a while ago when it seemed like almost every instructable required an expensive computer-controlled tool, and I wouldn't want such instructables to be rated poorly just for that. It would be similar for instructables requiring ownership of a sailboat, etc.

1.1.2 However, even a multi-dimensional five-star system would be susceptible to people voting at the extremes of the scales, as you mentioned. Therefore, I suggest substituting text-based scales in place of the stars. For example, the photo quality scale could have the following options: "All or almost all photos are clear and show all necessary details", "Most photos are clear, but some details are difficult to see", "Many of the photos are worth including, but some are not", "The photos add only a small amount of value/clarity", "This instructable would be no worse if the photos were removed". Each option would be given a point value of 1–5, but the point values would not be shown to the person submitting a rating, so that they would be more likely to rate honestly instead of just "1" or "5". Maybe the textual rating options could even be shown in random order. The point values would be used to calculate the average rating in that dimension (perhaps using that Bayesian algorithm).

1.1.3 It would be good to allow users to submit ratings in only some dimensions, so that they can still provide ratings even if they don't know how to or don't want to provide some of the ratings, or if one or more of the dimensions is not applicable.

1.2 Back to Wikipedia. The "Help improve this page" box asks readers "Did you find what you were looking for? Yes/No", and lets them write a suggestion for improvement.

1.2.1.1 The question "Did you find what you were looking for? Yes/No" or "Did this instructable help you? Yes/No" is fundamentally different from the quality ratings because it asks people whether or not they found the article/instructable useful, not how good they think it is. A usefulness score for each instructable (maybe number of "Yes" minus number of "No", or maybe use a Bayesian algorithm to compensate for low views) would be another way to sort instructables. It's somewhat similar to the old plus/minus system, but it's intended to measure just one aspect, usefulness, instead of condensing all aspects into one 'good' or 'bad', and it's probably easier for the users than picking from a range of usefulness values.

1.2.1.2 A similar suggestion feedback feature on Instructables might help new authors by letting other users submit suggestions for improvement without putting them in the comments section (which might not be monitored closely by the author—authors should be notified when they receive suggestions—and which would thereafter be more focused on providing helpful tips to other people, asking questions, and showing pictures of re-makes than on the original author's presentation). On Wikipedia, the suggestions and yes/no ratings are then displayed along with the names of users who submitted them on a feedback page, which, because Wikipedia articles are publicly editable, is visible to everyone. This raises the question of whether or not the feedback should be publicly viewable. Since an instructable is only editable by the author and collaborators, they're the only ones who actually need to see the feedback. On the other hand, making feedback public (not on a separate page, but beside the comments or below the info box, for example) would reduce duplicate suggestions. Wikipedia also allows editors to mark suggestions as resolved once the article has been edited accordingly.

1.2.2 Wikipedia only asks for suggestions after the "Did you find this article useful? Yes/No" question has been answered, but I think on Instructables they should be independent questions, so that people can submit suggestions even if they weren't actually looking for anything (in which case "Did you find what you were looking for?" would be meaningless) and so that people can say whether or not the instructable was useful even if they don't know how it could be improved (such as some very well-made instructables).

2. Comments

2.1 We NEED hierarchical comments back. Reddit has them for a reason. Instructables had them for a reason. They make it possible to have a coherent conversation with multiple branches that can easily be read at a later date and by other people. Even Facebook seems to have realized that flat comments make discussion difficult; they're starting to implement two-level comments like Instructables now has. It's better than one level, but only slightly. There are some blog CMSes that limit comments to four or five levels deep, and that's almost as bad because once you've gotten four or five messages into the conversation, it's difficult to continue without the subsequent branches getting mixed up. Now, on Instructables, it's difficult to have any conversation without the branches getting mixed up. Case in point: comments on this very post.

2.2 Let users publicly* mark comments that they think are helpful, insightful, or otherwise worth reading, and then allow users to sort comments by those ratings. This is what reddit does, and for the most part, it works well; the default comment sorting method there is based on ratings.** This would move comments that are worth reading closer to the top, where they'd be more likely to be read. Currently, the only way to express that a comment was good is to say so in a reply to it, which doesn't promote the good comment and only hurts the overall signal-to-noise ratio. Furthermore, I suggest incorporating Slashdot-style helpful/insightful/good-question/funny/etc. options*** (of which a user can select more than one) instead of just a 'like' or non-specific upvote, so that readers can sort comments by good questions to see if their question has already been asked, by helpfulness to look for any tips from other users they should take into account before starting a project, or by insightfulness if they're trying to find a way to improve upon the original project.

*I don't know whether the names of people who have given particular ratings to a comment should be public, but the total number ratings/total number of each type of rating a comment receives should be public; in my opinion, that's half the point of rating them.

**The only problem I see with sorting comments by rating by default is that new comments are unlikely to be seen. On reddit, threads don't usually last more than a few days, but on Instructables, I think the default sort should still be reverse-chronological, so that new comments get a chance to be read, replied to, and rated.

***I don't suggest incorporating any other aspect of the Slashdot moderation system. I think these rating options should be available to all users, because comments on obscure instructables would be unlikely to be seen by Slashdot-style moderators.

2.2.1 If you've found that worthless comments have been a problem—which I haven't—you could add another option, not worth reading, which would push that comment toward the bottom in all non-chronological sorting schemes. I feel that the wording of this option would be important, because many people on reddit use the downvote on things they disagree with, when it should be reserved for things that aren't worth reading. Maybe the "be nice" comment policy should include a point about appropriate downvoting if downvoting is implemented. On the other hand, the whole downvote/not worth reading option probably isn't necessary.

Summary of my suggestions: Multi-dimensional rating system for more clear and useful ratings, with textual scales instead of stars to promote honest and moderate ratings instead of just "1" or "5". Yes/no feedback on usefulness. Suggestions for improvement, separate from comments. The return of hierarchical comments. Ratings for comments and the ability to sort comments by rating.

P.S. Just tell Google that every recipe has five stars.
Kiteman Ian011 year ago
If the "dislikes" are meanspirited, so? That's the nature of the internet.

Other websites tolerate malicious negativity, do we ought to as well?

Sorry, that has never been the attitude here, and I and the rest of the Community Team put a lot if effort into making sure we do not descend into the cesspit of negativity that some internet users seem to think us an appropriate way to interact with the rest of humanity. This is not YouTube. A true Maker is positive and constructive.

If somebody cares enough to express their opinion through a complex, multi-optional rating system, then they care enough to post all that needs saying in the existing comments.
You're clearly missing the point, Kiteman. If someone "mean-spiritedly" votes 1 star on a person's 'ible, is it going to cause the site to "descend into the cesspit of negativity" that you describe? Of course not. Your extremism is neither necessary nor constructive.

If someone has a good 'ible, and someone gives it a negative vote out of "meanness," so what? It's not like it's a vulgar comment, and there's no knowing whether the vote was out of "meanness" or because they honestly didn't like the 'ible. With the CURRENT system, you give someone who's offended, or sincerely dislikes an 'ible only ONE way of showing their distaste:  Through a mean comment.  

Either way, if it's a "good" 'ible, then the positive votes will far outweigh the negative votes, and the end result is still the same.  The "mean spirited" individual got to throw his fit without having to devolve to making mean comments, and the ultimate impact to the site is nil.

Kiteman: "If somebody cares enough to express their opinion through a complex, multi-optional rating system..."
While someone else may have asked for a "complex, multi-optional rating system" it certainly wasn't me.  I asked for a simple like/dislike as has worked brilliantly for YouTube for about 6 years.

Kiteman: "...then they care enough to post all that needs saying in the existing comments."
Once again, you're utterly ignoring my previous statement.  Comments would be of absolutely zero use for any of the following (as I previously mentioned):

From previous post: ------------------------
"I've seen a number of sites like this go straight to hell after taking away any dis-incentive to post garbage.  With absolutely NO way for people to voice displeasure at having their time wasted even clicking on a lousy instructable that someone took ZERO time to produce, such as this:
http://www.instructables.com/id/wooden-sword-1/
or this:
http://www.instructables.com/id/cardboard-gun-prop/

Soon the site will be so full of absolute garbage, with no way to jump to the "highest rated" (and no, "most liked" is certainly not the same thing) that those who DO take days, weeks, or even months to create a thorough and impressive instructable will lose all incentive to participate.  Why would they want to waste their time if their 'ible is guaranteed to get buried beneath the ego stroking of 10 year olds?  And if I'm coming to search through the 'ibles, I'm going to be looking for those that ARE of the best quality, without wanting to have to click through and read several dozen to find the one that some effort was put into.
...
A [YouTube-Like] system only makes sense on a site like this where you may find two dozen different 'ibles on composting and wish to quickly find those that were the best designed, not just the ones that happened to be on the site for the longest, or that were posted by people who have 500 facebook friends."
------------------------------

This site was not (when I became a Pro member, at least) here for the sole purpose of allowing people to stroke their own egos, or to cater to pre-teens.  It was here as a resource for people who like to make stuff themselves share their knowledge and expertise, or to learn from those that have the knowledge or expertise.  Perhaps as Instructables as moved into the more commercial realm, the charter has changed.  If the goal is to placate everyone in order to drive up user counts in order to make as much money as possible, let's be honest about it at least.

By removing the rating system so as not to "hurt anyone's feelings" you've lowered the bar to the point that this is just another site where anyone can post anything (as evidenced by the above links), and trying to filter through the trash to find anything of value will soon become more trouble than it's worth.

This said, Kiteman, your post has made it clear that you've little interest in my opinion.  I'm sure this will quickly devolve to, "Well, if you don't like it, leave!" so this will be the last I have to say on the subject.
Perhaps a "needs improvement" button should exist along with a "like" button. It gets the same point across (imo) in a "be nice" sort of way, and lets the author know that something is lacking (plus tells other members that the project isn't quite up to par).
Unfortunately if you have a 'needs improvement' button you will then need a how to improve comments box otherwise it wouldn't work. There's nothing worse than going to an audition and not getting feedback as to why you weren't picked which is the same as why does this instructable need to be improved.....and who is the authority recommending this? Needs more thought.
Woah - I hadn't even addressed a comment to to you, and you've managed to manufacture an entire row, all the way to putting words in my mouth and storming off in a temper at those words.

OK, you've had your last word, but in case other people are interested in my "extremist", fact-based view;

Down-rating has never prevented the posting of rubbish (see the very YouTube and FaceBook you wish to emulate). What has reduced the posting of rubbish has been the positive response to good projects, and the hours and hours of work Staff and the Community Team put in behind the scenes every day to helping people to improve their projects. Members who post poor projects without any interest in improvement quickly lose interest because of the lack of views and the lack of positive comments/ratings. Their projects slide into obscurity, only seeing the light of day when somebody drags them up to "prove" that loads of rubbish is being posted and drowning out the good stuff (which is nonsense, as any quick browse of the "recent" lists will show).

Historically, the ability to down-rate projects has encouraged trolling, with members posting low-quality projects, then using multiple sock-puppets to vote them up and to vote other, competing projects down. It got to the point where down-votes had to be excluded from contests where entries were selected as finalists on the basis of popularity, because the only projects that were coming up with positive totals were the dross posted by the trolls.

Now, you are welcome to think whatever you like (and you clearly do), but the above is posted from direct experience, and is a matter of record, not opinion.
"Woah - I hadn't even addressed a comment to to you, and you've managed to manufacture an entire row, all the way to putting words in my mouth and storming off in a temper at those words."

No, you'd only quoted me then proceeded to put words in my mouth and argue against your straw-man. (Projection much?)  That you quoted me and then responded to the wrong thread is hardly my fault.

"Down-rating has never prevented the posting of rubbish"
Once again, you conveniently ignore statements that have already clarified this.  At no point did I say it prevents this.  I've (repeatedly) stated that this would allow us to easily filter through the rubbish, and continue to stand by that statement.

"What has reduced the posting of rubbish has been the positive response to good projects"
Which is why I was very easily, and quickly able to show two (new) rubbish posts that I found while trying to find a good post on how to use cardboard to build props - posts that wasted my time as I had no context to know that they were rubbish without opening them.  Coddling people has never, in the history of the world, stopped anyone from slacking off.  

"Historically, the ability to down-rate projects has encouraged trolling, with members posting low-quality projects, then using multiple sock-puppets to vote them up and to vote other, competing projects down."

Yes, and I've been the "victim" of this exact type of trolling on YouTube.  Do you know how much it impacted the overall net result?  Zero.  Zilch.  Nada.  It's a failed argument from the start.  Even moreso if you limit the ability to vote to those that have Pro memberships.  A quality 'ible will overwhelmingly get enough positive votes to make such trolling pointless, and virtually non-existent when the "right" to vote is limited.  (This is posted from direct experience, and is a matter of record, not opinion.)

"It got to the point where down-votes had to be excluded from contests where entries were selected as finalists on the basis of popularity, because the only projects that were coming up with positive totals were the dross posted by the trolls."

Cool story, bro.  Still, it has nothing to do with the conversation at hand.  The contests already have their voting completely separate from the rating system that would be used for said filtering.  

As I do business metrics for a living, I can assure you that there are patterns that can be discerned any time someone may attempt to "game" the system that are easily thwarted.  This is a matter of both record and professional opinion.

"Now, you are welcome to think whatever you like (and you clearly do)"
Thank you for your permission.

Back to my original post - it was simply to state my (very strong) opinion that a like/dislike option is necessary for a site such as this.  I was not attempting to get into this "flame war" but I will certainly defend my position, particularly when it's responded to in such a dismissive and condescending manner by one of the very people claiming that this site should have a strict "be nice" policy.  This is a matter of record and opinion.

And yes, I did say I wouldn't respond again, but God forbid I be accused of "storming off in a temper."
What flame war? The only one flaming here is you, but, you can spout and fizz as much as you like, you are not going to change anything on this site by TL:DRing at me.

(Oh, and please pay attention - when I mentioned "trolling", I meant here, not YouTube. Do you see it now? No. Do you know why? Because we took away the incentive to troll by removing the ability to down-rate. This is not YouTube, this is not FaceBook, and I, personally, am proud of that fact.)
I totally agree with you, let's keep this a positive site! We have enough 'crap' in the world without adding to it!
Yeah, no true Scotsman is negative! :P
@Lira

;-p
Ian01 Kiteman1 year ago
I agree with everything you said, but it seems to me that you might not fully understand what I meant. (Of course, I understand that you're busy and probably don't have time to respond to everything.) I'll try to clarify:

Other websites tolerate malicious negativity, do we ought to as well?

I carefully chose the words "not worth reading" to mean exactly that, and to discourage malicious use (though it would be impossible to prevent malicious use). I did say that a downvote option probably isn't necessary. I didn't mean to advocate for it. I just included it for completeness. Feel free to ignore it while reading the rest of my suggestions; it has no bearing on them.

A true Maker is positive and constructive.

I agree. To clarify my intentions, the downvote/"not worth reading" vote would serve to reduce the prominence of comments that are not positive and constructive. (I personally think a lot of people don't want to report comments just for being unhelpful, because it feels like censorship or wasting the team's time.) Again, though, I don't think it's necessary, and I just included it for completeness.

If somebody cares enough to express their opinion through a complex, multi-optional rating system, then they care enough to post all that needs saying in the existing comments.

True, but there's a difference: computers can understand ratings, but they can't understand comments. Ratings like I described would allow people to search and browse based on rated end-product quality, writing quality, etc. When I search for something specific (and I try to search thoroughly), I usually get a lot of low-quality, unhelpful results that just get in my way and waste my time. Even just separating the presentation quality and product quality into separate ratings might help with that.

(Tangent: I hope it's not the case, but it seems like you and/or others might think that instructables should not be ranked or compared numerically. If you don't think that, you can ignore the rest of the paragraph. If you do think that, I strongly disagree. Some are just better than others, and we shouldn't pretend otherwise. LED Throwies is unarguably better than BLOW UP A GLASS SNAPPLE WITH YOUR HAND!!!!!, for example.)

So, to sum up, section 2.2.1 can be completely ignored, and complex ratings do have some advantages over comments. I hope this has cleared up my intentions on those two points. :)
wotboa1 year ago
I agree that most ratings systems on the web have become meaningless. Your five-star method works well as a project climbs toward 5, but you realize that most projects are rated over 4.5 with, as you say, the second decimal point deciding the tie-breaker. (At what, 0.2% full scale?)

If viewers are inclined to give any project a rating, encourage them to do more than simply push a button. In other words, discourage the lookie-loos who have no opinion. Any meaningful rating should be thoughtful and complete.

Ian01 posted a comment Nov12:
"I suggest the following dimensions: quality of finished product, inclusion of proper techniques, writing quality, photo quality, completeness, ease of following instructions, innovation, accessibility*, etc."

Ian01 has made a great preliminary list (5 stars!). Take a poll to get a consensus of what facets viewers might think are important for a project to receive a high rating. With six or eight or ten categories, the viewer who truly wants to give a rating will take the time to do so. The fly-bys will opt out and move on.

Maybe you can out-wiki Wikipedia and start a new trend. Make ratings meaningful, not just push a button to vote. (Then sell your new 'ratings software' to other sites wanting to use it. With embedded links to Instructables, of course. Heheheheh.)
wotboa wotboa1 year ago
I made a short list of some of the categories that I use before starting my own projects.
Perhaps the list will spark some interest in creating a new 'category' rating system.

Parts/material availability
Tools required (related to average home workshop)
Skills required to finish the project
Cost (time and money) versus usefulness
Complexity versus usefulness
Difficulty versus usefulness
Safety - assembly techniques
Safety - materials
Safety - finished product

On the Instructables project page also rate:
Written instructions
Photo quality
Innovation of design
Level of difficulty
I want to build this
I was just looking at my YOU page under "Following" and I'm now confused...

If we removed the Star Rating System, why are there NEW projects showing a Rating that were published AFTER the site changes? Some have ratings, some don't. I thought it might have something to do with the number of times the projects were selected as "favorites", but that can't be the case because some Instructables with NO Rating have been favorited...

Any idea why or how some projects are getting a rating without a rating system???
Jayefuu1 year ago
I saw this on the oatmeal today. I thought it relevant to some people's comments about the new lack of allowing giving negative ratings.

14.jpg
Haha, I haven't Oatmeal in ages!
I really like the idea of the "I made this" and "want to make this" buttons.  That sounds like it would be highly complimentary to authors, and would allow me to give a little more feedback than just favoriting something.  How about a tribute page for each instructable, where members can post pictures of their own variations and attempts?  I've been impressed with some of the variations given in the comments of (for example) MrBalleng's instructables, but going through a couple hundred comments to find pictures is a little daunting.  Also, don't use "it" maybe so it doesn't sound like braggin' or "missed opportunities" on Craigslist.  Or maybe do.  I could see people being very entertained by hitting the "I want to make it" button over and over again.
I would love a "want to make it" button.
bajablue1 year ago
I'll get used to it. ;-)
I miss "Project Of The Day".
The Project Of The Day is still alive! You can see it by hovering your mouse over the Explore tab up top.
Thanks mikeasaurus, that made my day!
Me too!!!
makendo1 year ago
Glad to see the end of the rating system. The addition of favourites is a nice touch. A metric for non-members like FB shares or tweets might be useful also (I don't know if you even have this data, but it at least allows would be raters to choose their own brand of endorsement).
How about a +/- rating system in which the displayed rating will never go below zero? For example:
How to Build an Ark gets 3 upvotes and 7 downvotes. The rating would read "0" until the invisible downvotes are canceled out by more upvotes.

OR

A +/- rating system in which the rating shown is a colored bar(just like YouTube), but grows in thickness with each rating until it hits a maximum thickness(say, 50 votes). People would instantly know the rating(Percentage, NOT weighted) of the instructable and the validity of the ratings.

OR

A +/- rating system in which a user will be required to give an explanation for downvoting, presented by a list of checkboxes.

Ideally we would use YouTube's rating system.
If lots of people think my instructable is shit, I want to know about it with absolute clarity so that I can improve upon myself. Negative reviews are extremely meaningful, and hiding them doesn't help anyone.
strods1 year ago
I'm fine with the favorite and agree with the reasoning. It would be nice to have a talley of "I made this". if we have 10 "Made" besides the orginal author and the comments are all fair or positive, then the chances are it is legit. But if there are a lot of comments or recommendations then maybe the 'ible wasn't the best.
Kiteman strods1 year ago
An "I made this" button is a regular request, and one I heartily support.
I feel strongly about the I made this button. Currently, Instructables is not about making per se so much as inventing or creating; only something that can be made into a *new* instructable will really give you something to talk about/share on the site. To me, it would be just as awesome to have a stat on how many instructables you have recreated, recognizing making - not inventing - as the true activity driving the site.
+1
(Although I predict, on that metric, that recipes would become the main focus of the site...)
Well, not if you had some kind of display system that specifically had "areas" rather than just showing the top X instructables in all places. I'm having trouble describing what I mean here; maybe I'll be better able to explain in the morning...
Oh, I know, but somebody might look at the site-wide numbers and decide that the members want it to be a cookery site...
I mean, how's that different than if you look right now at site-wide numbers of published instructables and count which are recipes and which are anything else? My understanding is that it's not even close.
I mean, "folk", especially people brought by google, are more likely* to recreate a recipe than many other kinds of project. If they started an "I made this" button**, that data taken alone might skew things towards food.

* No evidence, just assumptions.
** An option I still support.
Oh. I thought only registered members could rate/like/whatever, so I was envisioning the same towards the I Made This button.

only registered members

Yes, that's how I see the 'I made this' feature, too. Also, it should generate two lists:
'who made this' on the I'ble, and 'what did I make' on the member profile.

On a different note:
This new comment system is pretty hopeless if you don't quote at least a tiny bit. A quote button (or a feature that works by selecting some text before you hit reply) in the default reply mode would be useful.

>This new comment system is pretty hopeless if you don't quote at least a tiny bit.

Ahem...guilty as charged...it's such a pain that I keep forgetting. :p We HAVE to either have the indentation fixed, or have a quote button.
This new comment system is pretty hopeless if you don't quote at least a tiny bit. A quote button (or a feature that works by selecting some text before you hit reply) in the default reply mode would be useful.

How are you doing your quoting Moem? I've been tabbing and italicizing. Yours looks cooler.

How are you doing your quoting Moem?

I'm using the button with the double quote symbol, next to the bulleted list button, in the rich text editor for comments.

I don't have such a button. I guess having your conversations understood is now a Pro feature.
Ian01: "I don't have such a button. I guess having your conversations understood is now a Pro feature."

Oh man this is rich! Thowback to another ill-received site change.
We really do need the old hierarchical comment system back. This is getting ridiculous. Is the team just changing things for the sake of change? I only recently got over the last one. 
Land01: You don't have a Rich Editor button below the comment box? Are you sure?
Land01: You don't have a Rich Editor button below the comment box? Are you sure?

If I remember correctly the rich edit box is a pro only feature.

If I remember correctly the rich edit box is a pro only feature.

Ah, bummer. Thank you.

Ahem...guilty as charged...

It's really not your fault. It would be so much better if quoting were easier.

To me, it would be just as awesome to have a stat on how many instructables you have recreated, recognizing making - not inventing - as the true activity driving the site.

I second this. Quite a few of my Instructables have comments with photos of people that have done similar after seeing my Instructable. Short of adding a step manually with photos and comments from others, there's no way for others to see at a glance who's made it.

To me, it would be just as awesome to have a stat on how many instructables you have recreated

Yes... because clicking a star is nice, but actually making something, that's what it's all about!
And having a stat on how many times your instructable has been recreated by others would be the ultimate form of recognition.

Exactly! (I accidentally a clause in that sentence - I mean, it would be just as awesome to have X amount of recreated instructables on my profile as having X amount of published instructables.)
I like it so much, I often give a patch to people who recreate my projects, sometimes even a pro-code.
dogsgomoo1 year ago
There needs to be some mechanism to mark Instructibles that have issues with them, other then a general "flag" button. Especially based on some of the very iffy "featured" ibles that have recently appeared in the newsletters. Some had plagiarized photos and content without sources!

There has to be a way to separate out the "Student Presentations" from the Instructibles made by people who engaged in the creation process and shared the fruits of their labors to an audience who is then free to expand upon and innovate.

An" always be nice" policy is well intentioned but you only have to look at Etsy to see how it can become a very heavy thorn in community content, discussion, and growth. Forced positivity is not a good thing.

Why not something like the Reddit +/- mechanism?
Sounds like a good approach.
Nice to hear you take in account not everyone uses Facebook.
It's just a rough draft, but what do you think?

[GorillazMiko approves] [thumbs up emoji]
I don't like the word "favorite", it kind of means "bookmark" in some context. I'd like to see a "upvote"/"like" button, plus a separate "favorite" button.
Javin0071 year ago
We definitely need a "dislike" button as well. (Facebook needs to learn this, too.) For instance, if two instructables have 10 likes, but there's nothing to compare it against, they would seem "even". But if 190 people disliked one, and nobody disliked the other, then that says something.

I've seen a number of sites like this go straight to hell after taking away any dis-incentive to post garbage.  With absolutely NO way for people to voice displeasure at having their time wasted even clicking on a lousy instructable that someone took ZERO time to produce, such as this:
http://www.instructables.com/id/wooden-sword-1/
or this:
http://www.instructables.com/id/cardboard-gun-prop/

Soon the site will be so full of absolute garbage, with no way to jump to the "highest rated" (and no, "most liked" is certainly not the same thing) that those who DO take days, weeks, or even months to create a thorough and impressive instructable will lose all incentive to participate.  Why would they want to waste their time if their 'ible is guaranteed to get buried beneath the ego stroking of 10 year olds?  And if I'm coming to search through the 'ibles, I'm going to be looking for those that ARE of the best quality, without wanting to have to click through and read several dozen to find the one that some effort was put into.

YouTube's "like/dislike" rating system is a tested and proven system.  Yes, I understand that some people may get their feelings hurt when only 10 people vote on their submission making the results potentially skewed, but that's the nature of any like/dislike system.  When I'm looking through YouTube videos for something funny, and it says "Funnyiestest!!!!111oneone" in the title, the first thing I look at is the rating.  If it's 90% to the negative, I know not to waste my time. 

A similar system only makes sense on a site like this where you may find two dozen different 'ibles on composting and wish to quickly find those that were the best designed, not just the ones that happened to be on the site for the longest, or that were posted by people who have 500 facebook friends.  (I have 3.)  The argument that people "were mean" in marking it to the negative just doesn't hold water.  And the only other reason for getting rid of the plus/minus system is because people "wanted a system that was more nuanced..." So the answer was to REMOVE THE SYSTEM ALL TOGETHER?  SERIOUSLY?

So yeah, I'm firmly in the camp that just flagging something as "liked" or "favorited" isn't nearly enough.  If the "dislikes" are meanspirited, so?  That's the nature of the internet.  It's also easily thwarted by simply not showing the "rating" at all until a set number of votes have been placed. 

I'll also second (third, fourth, or fifth, I haven't been keeping count) that a series of additional ways to bookmark the 'ibles would rock.  Perhaps give people the ability to create their own groupings, or have some default bookmark groups (ie: for review, in work, must build, etc.)  I currently do something similar with my own browser bookmarks, but of course those are kept locally, so it's a bit of a hassle if I'm burning time at work browsing the newest 'ibles, and marking those I'd like to read through more thoroughly later.

K, so that's probably considerably more than my 2 cents, but I have change.
I rated this topic 5 stars.
ewilhelm (author)  Lithium Rain1 year ago
Me too. You can also still rate Instructables in the Food category until we figure out how we're going to calculate the rating passed to Google.
LOL me three! :) What if in the food category you had a rating system with a review button, so the starts have meaning, averaged with a short system of 5 questions and then comments if wanted. Then a user who makes it could write and rate. I feel those who make items in the food catagory and are active instructable users would have no problem coming back after making it and rating. Just a thought.
Hi Eric,
I would like to say that i have enjoyed this site since i joined . Some of the changes seem like its making it harder to view instructables as a whole. I mean i come on this site a few times each day . I liked seeing how far i moved up the ladder with view counts and seeing where the views came from. Right now it seems so limited when i pop on the sire whether its because its new or maybe it is a little limited now for what you can do.
Hi Eric,
I would like to say that i have enjoyed this site since i joined . Some of the changes seem like its making it harder to view instructables as a whole. I mean i come on this site a few times each day . I liked seeing how far i moved up the ladder with view counts and seeing where the views came from. Right now it seems so limited when i pop on the sire whether its because its new or maybe it is a little limited now for what you can do.
Oh, burn!

;-D
poofrabbit1 year ago
I am fine with the star system gone and like seeing how many individuals had favorited something. I agree with others, some way to sort our favorites would be great. I find myself sometimes deleting things out of my favorites, not because I like it any less, but because I have to many. I feel if I had a way to sort I would be much more likely to go back and create the projects I like, which would be all around awesome. This is why I created some guides, but even then you get so many things on the guide it becomes hard to navigate through, might be nice if the guides could have pages of sorts, or tabs that things could be sorted under, same with faviroates, just a thought to throw out there. As always, I appreciate the experience you are giving to all of us and such a creative space to play!
ehudwill1 year ago
I like the idea of taking the star ratings away. I can see your point in using it and then removing it. I like the favorite and don't think we need any other way to show if people like the instructable.

What I would like to see is a way to organize the instructables I have favorited. I would like to have a way to search them and put them together. I have tried the guide feature and it just does not work the way I want it to.
JDTagish1 year ago
For me, I use the favorites as a way to "remember" an Instructable that I would like to try in the future, or had an element in it that would help me in a project that I am currently working on or may work on. I wouldn't use it as a way to "rate" an Instructable because then I would have no way of sorting through things that were interesting to read, or that I liked, but wouldn't necessarily build/make for myself.

IMO, the "Favorites" needs to remain a way for users to choose Instructables that have a meaning to them, and not as part of a "rating" or "liking" system. I completely understand why the starred rating system didn't work out the way that you wanted to. It seems to me that a "Like" or similar button would accomplish what you're looking for without users losing the ability to keep track of the Instructables that offer them a benefit or reference on a project.

As I see it, the alternative is to allow users a way to "store" Instructables (Possibly with notations - "for garage" or "need to do this!") to be able to keep track of the Instructables that have meaning or value to them, and then keep the "Favorites" as a rating method.

But, as it stands now, I won't use the Favorites the way you're intending because then I'll lose track of the Instructables that I want to be able to refer back to in the future. As a PRO member, I can of course, download PDF's of the projects that interest me, but I imagine that free memberships without this ability use the Favorites in place of PDF downloading for tracking.
Mrballeng1 year ago
Are "stats" going by the wayside? It was good info to see which websites were referencing your work. Is there still a way to see that?
look at Randofo's comment http://www.instructables.com/community/changes/?comments=all#CW4WLEHH97JYHO7
Lorddrake1 year ago
I understand why the star system has gone the way of the dodo ... But I i have to agree with ElvenChild .. we need an "I think this is awesome button" in addition to the "favorite" button.

I don't know about other people, but I use the favorite feature to mark IBLES that I;
1) have completed
2) plan on attempting to recreate in whole or in part
3) want to share with people I know

There are alot of IBLES on the site that I do not have interest in trying to duplicate (don't want to add to my favorite list), but they are well put together and informative and I would still like to give the author kudos for a job well done.

just my 2 cents on the subject.
Very good point.

I don't want to fill up my favorites either just to acknowledge that an instructable is good. I use favorites like a temporary bookmark, and then once I've downloaded it, more often than not I remove it as a favorite.

I would prefer an alternative system to just show appreciation, such as a "thumbs-up" or a "Like" option.

It also can be misleading if an Instructable has 10,000 views but only 5 favorites. Maybe it was downloaded as a PDF 8,000 times!? But you would never know that the Instructable was so well received if you only have a mediocre favorite stat to go by.

+ 2 cents. :-)
ElvenChild1 year ago
On the subject of likes and favorites, for several years I have been trying to favorite various instructables... The problem is I have only succeeded once... And that was unintentionally (thats not to say that it wasn't a good instructable though). I think there should be a favorite button, and whenever it gets pressed, instead of showing a rating in stars, it should say something along the lines of "x people favorited this"
You're in luck, this is exactly the system that was rolled out in this release!
You can see how many people "favorited" a project right next to the authors name and number of views that project has, on the right hand side. You can add to this number by clicking the "favorite" button at the top of your page (the "favorite" bar now scrolls with the page at the top so you can access it at any time while browsing), these favorites will also appear in your YOU page.
I have to say I love this new "favorite" feature- it's far more valuable to me than a star rating system.
Yeah, I kinda sorta realized that almost right after I posted that... I must have misunderstood something while reading. Thanks!
+1
Or maybe it should say "x people thought that this instructable was totally, for the love all things robot (or maybe it should be diy) awesome"... Or somehthing like that anyway.
sarawelder1 year ago
what a great site this is. Thanks for explaining all those ratings and hidden agendas on Facebook! I, too, would love to be able to peruse my own favorites. At the moment I have to copy them to a special folder on my computer.
rimar20001 year ago
Always I said "Instructables is a very well managed page. I worked many years in system analysis, hence I know the difficulties the matter has.
I used favorites more than stars anyway. If it's something I find useful, I want to bookmark it for future reference. The only time I ever rated something was of it was really standout exceptional.
i "like" it. i think that system with absolute numbers is a much better statement about the quality/popularity of each instructable inside this community because the scale is open towards the upper end and not limited by that relative value of "5 stars".

Maybe i missed it but how did you transform these old "stars" into that new "favorites"? Everyone who voted for an instructable weather it was 1 or 5 stars is now favouriting this instructable or how does it work?
PKM1 year ago
Thanks for taking the time to explain this- I would never have twigged that Google's own star rankings would interact badly with our own, and all the subtle ways that negative ranking can act badly.

As a web privacy advocate I detest the sneaky use of the facebook like button in tracking user's behaviour, and I'm very glad that the site has its own mechanism.  Good work :)

Finally, if we're blue-sky thinking about rating systems, I've always thought it would be nice if you could differentiate between great projects, and great writeups.  To be featured you have to have both, but I'm sure there are a lot of hidden gems, either incredible things people have made which are let down slightly by poor writing, or simple makes which are nevertheless funny, or tell a great story.  I think Slashdot's Informative / Insightful / Funny tagging is probably the best instance of this I've seen, and I don't know how it would translate to Instructables, it's just food for thought.
Lorddrake1 year ago
thanks for clearing that up .. now that you have explained it, I understand what you want to achieve.
Hack42Moem1 year ago
Thank you. I love this kind of explanations. It makes it so much easier to understand the reasons for changes, whether we like them or not. This is a Good Thing.
kelseymh1 year ago
Thank you very much for the details, Eric! For those of us who've been around for a while (or maybe just for science geeks :-), this kind of public analysis is quite welcome. When something is changed, it can (I think) help to defuse the typical "oh it's different, I hate it" to understand what was wrong with the old system, and why the new system might turn out to be better.

As to the specifics: Grade inflation is just sucky. Thank you for getting rid of it! Facebook is sucky. Thank you for not simply sniffing their butts :-) Letting people vote for something if they enjoy it, and basing "ratings" on how many people did so, seems entirely in line with I'ble's historical "be nice policy." I hope that it turns out to be productive in the long term. Woo hoo!
+1
iceng kelseymh1 year ago
+1
iceng1 year ago
Besides using views as a guide to popularity and all that entails, the count goes up
dramatically on being featured and more so when front_paged.

I use the favorite category of instructables that I want to review later,
some very interesting ibles and a bunch that help me answer questions.

It is difficult to keep over two hundred favorites ( takes too long to search through them ).
I'm constantly culling them back and loosing the battle.

I would really enjoy a search feature of only my favorites.

A

 
Great idea! I love it!