Crowdsourcing and Peer Production I


May 8, 2012 @ 14:30, Room: 19AB

Chair: Mira Dontcheva, Adobe Advanced Technology Labs, USA
Communitysourcing: Engaging Local Crowds to Perform Expert Work Via Physical Kiosks - Paper
Community: engineering
Contribution & Benefit: Introduces communitysourcing: the use of physical kiosks to target existing crowds of expert workers with specific large-volume microtasks. Demonstrates through a deployment that communitysourcing can successfully elicit high-quality expert work.
Abstract » Online labor markets, such as Amazon's Mechanical Turk, have been used to crowdsource simple, short tasks like image labeling and transcription. However, expert knowledge is often lacking in such markets, making it impossible to complete certain classes of tasks. In this work we introduce an alternative mechanism for crowdsourcing tasks that require specialized knowledge or skill: communitysourcing --- the use of physical kiosks to elicit work from specific populations. We investigate the potential of communitysourcing by designing, implementing and evaluating Umati: the communitysourcing vending machine. Umati allows users to earn credits by performing tasks using a touchscreen attached to the machine. Physical rewards (in this case, snacks) are dispensed through traditional vending mechanics. We evaluated whether communitysourcing can accomplish expert work by using Umati to grade Computer Science exams. We placed Umati in a university Computer Science building, targeting students with grading tasks for snacks. Over one week, 328 unique users (302 of whom were students) completed 7771 tasks (7240 by students). 80% of users had never participated in a crowdsourcing market before. We found that Umati was able to grade exams with 2% higher accuracy (at the same price) or at 33% lower cost (at equivalent accuracy) than traditional single-expert grading. Mechanical Turk workers had no success grading the same exams. These results indicate that communitysourcing can successfully elicit high-quality expert work from specific communities.
LemonAid: Selection-Based Crowdsourced Contextual Help for Web Applications - Paper
Community: user experience
Contribution & Benefit: We present LemonAid, a new approach to help that allows users to find previously asked questions and answers by selecting a label, widget, or image within the user interface.
Abstract » Web-based technical support such as discussion forums and social networking sites have been successful at ensuring that most technical support questions eventually receive helpful answers. Unfortunately, finding these answers is still quite difficult, since users� textual queries are often incomplete, imprecise, or use different vocabularies to describe the same problem. We present LemonAid, a new approach to help that allows users to find help by instead selecting a label, widget, link, image or other user interface (UI) element that they believe is relevant to their problem. LemonAid uses this selection to retrieve previously asked questions and their corresponding answers. The key insight that makes LemonAid work is that users tend to make similar selections in the interface for similar help needs and different selections for different help needs. Our initial evaluation shows that across a corpus of dozens of tasks and thousands of requests, LemonAid retrieved a result for 90% of help requests based on UI selections and, of those, over half had relevant matches in the top 2 results.
Is This What You Meant? Promoting Listening on the Web with Reflect - Paper
Contribution & Benefit: Observes that listening is under-supported in web interfaces, explores the consequences, and contributes a novel design illustrating listening support. Field deployment on Slashdot establishes potential of this design direction.
Abstract » A lack of support for active listening undermines discussion and deliberation on the web. We contribute a design frame identifying potential improvements to web discussion were listening more explicitly encouraged in interfaces. We explore these concepts through a novel interface, Reflect, that creates a space next to every comment where others can summarize the points they hear the commenter making. Deployments on Slashdot, Wikimedia's Strategic Planning Initiative, and a local civic effort suggest that interfaces for listening may have traction for general use on the web.
#EpicPlay: Selecting Video Highlights for Sporting Events using Twitter - Paper
Contribution & Benefit: Explores differences between crowd-sourced (through Twitter) video highlights of broadcast sports compared to nightly sportscast highlight reels. Illustrates utility of separating home and away tweets.
Abstract » During a live sports event, many sports fans use social me-dia as a part of their viewing experience, reporting on their thoughts on the event as it unfolds. In this work, we use this information stream to semantically annotate live broadcast sports games, using these annotations to select video high-lights from the game. We demonstrate that this approach can be used to select highlights specific for fans of each team, and that these clips reflect the emotions of a fan dur-ing a game. Further, we describe how these clips differ from those seen on nightly sportscasts.