First of all, thanks to all for entering, voting and sending us feedback on the Cartoon Caption Contest, we’re always looking to improve.
We’ve had some questions/concerns/kvetching about the voting process. Why does it take so long to see all the captions? Is it fair? Can’t you do it for us and give us three to pick from?
Well the answers are because there are a lot of captions, yes it’s fair-ish, which is better than unfair-ish and no, no, no a thousand times no (actually two-thousand-three-hundred and fifty-seven times no which was the number of captions we received in the last contest).
Why you ask, can’t we weed through and select a handful for you? Because we answer, based on scant research but a lot of experience, whenever a person is presented with a large sample of “humorous” content, your humor-meter gets maxed out. Multiply that by the fact that you’re looking at generally pretty similar captions and before too long nothing seems funny, or everything seems funny and you go barking, ha-ha-ha-woof, mad. That’s why we use crowdsourcing. With a larger pool of humor assessors, the results become more accurate and the potential to go humor blind drops.
Onto fairness: The voting application is randomized, so there is no set order to the captions. What you see as the first 20 won’t be what others see, because that WOULD be entirely unfair. There is actually some pretty sophisticated calculations developed with a team at the U of Wisconsin happening behind the scenes to ensure that all captions get distributed equally at the start of the voting timeframe, then as the votes pile up, the cream rises to the top and the craptions sink to the bottom. The more highly voted captions get more views to enable to more popular entries to get a statistically significant number of votes to move us in the direction of the winner. Even with crowdsourced voting, ultimately it comes down to us picking the winner out of the upper group of 20 or so (which is mentioned in the rules). Sometimes the winner IS in the first place, sometimes we select a different high-ranking caption for a variety of reasons such as originality or good caption writing or both. So, between two equally rated captions, we will tend to favor the less frequently occurring joke. And, between two equally rated captions with the same gag, we will go for the one that we think is the better-written joke.
Here comes the science!
The application we’re using was developed by Robert Nowak and his team at the University of Wisconsin, the same team that developed the system the New Yorker uses. The algorithm for the caption rating system automatically focuses on the most competitive captions. Based on all the ratings received, the algorithm determines which captions have a statistically significant chance of winning and then selects one of these at random to collect a new rating. This is done each time a person is asked to provide a new rating. For example, a caption that has received two dozen ratings, all 1-star, is unlikely to be rated again. This winnowing process allows us to obtain more ratings for the most competitive captions and thus make a more confident decision about the winner. Conversely, the mathematics supporting the algorithm guarantees that there is very little chance that a truly “good” caption (as judged by the crowd) will not end up high in the ranking at the end of the process.
- We don’t intend for people to rate all the captions, nor do we need this level of effort. Rating a small number of captions (say <=100) will be very helpful and sufficient
- It is quite likely that a person will not see the caption they submitted, due to the randomization of the process
- The algorithm guarantees with high confidence that the most competitive captions will be rated adequately. There is very little chance that a truly “good” caption (as judged by the crowd) would not end up high in the ranking
End of Science lecture. Class dismissed.
Best,
Bob
P.S. All this said, this is all far from a perfect system. First, fluky things can occur. If they couldn’t we wouldn’t even have the word “fluke”. So, while it is very unlikely that a great caption is going to get 12 one-star ratings by chance right off the bat, could happen. Also, if you want to spend your voting energies trying to downgrade your best friends caption and boost your own, you might, by dint of malicious effort, be able to do that–but no more. New improved algorithms, don’t you know.