Cette page appartient aux archives web de l'EPFL et n'est plus tenue à jour.
This page belongs to EPFL's web archive and is no longer updated.

Human Computation Papers

Please submit a short description of the paper you discovered on human computation. Include the full bibliographic reference as well.

Posted by Katherine Skirving Larson on Wednesday 25 April 2012 at 16:31
Comments
This is a test.
Posted by Katherine Skirving Larson on Thursday 26 April 2012 at 16:30
PAPER:
Severin Hacker and Luis von Ahn. 2009. Matchin: eliciting user preferences with an online game. In Proceedings of the 27th international conference on Human factors in computing systems (CHI '09). ACM, New York, NY, USA, 1207-1216.

COMMENTS:
The work described in the paper is very interesting!
Similar as ESP, this is about designing a game which actually contains a concept of human computation inside. The game described, Matchin, provide a user two images and ask "which image do you think preferred by the other player?" Coupled with sigmoid score function, the authors make this game very interesting.

But the work itself actually has more to offer than that. Using the data retrieved from all games played so far, it is possible to construct a global ranking through all of this images. In some sense, this is a kind of social choice. Both of the work, an attempt to construct a global ranking and social choice theory, in my opinion could get some advantage from each other since the main focus here is to construct a global preferences over some candidate (or images).

Another important insight: On repeated tasks, when an incentive play a role, human do learn from previous experience and aim solely for more incentives rather than perform their true task. This phenomenon has to be taken into account on any mechanism when the main objective is to obtain user's true preferences, i.e. careless incentive design can be misleading.
Posted by Tri Kurniawan Wijaya on Friday 27 April 2012 at 7:10
Paper:
Y. Zhang and M. van der Schaar, "Reputation-based Incentive Protocols in Crowdsourcing Applications," IEEE Infocom 2012.

Comments:
This paper describes incentive protocols in crowdsourcing application using a model based on repeated games. Basic idea is to integrate reputation systems into existing pricing schemes used in crowdsourcing. The problem with pricing schemes is that it is not clear when requester should pay worker for the task they agreed on: if it is before the job is done, than worker might not have an incentive to work on it at all; on the other hand if it is after, then the requester might not want to pay for the solved task. Therefor the paper combines a payment scheme with a crowdsourcing protocol, and analyzes such mechanism using repeated games environment. The main focus of the paper is put on designing appropriate social norm in order to punish workers deviation from the selected social strategy. Authors prove that it is possible to design social norm based protocol which would prevent the effect of free riding, i.e. force workers make an effort and do their jobs. Although it is neglected in the proposed protocol, authors also analyze, using numerical simulation, the problem of requestors' not being truthful. Both theoretical and practical analysis discuss the relationship between the designed protocol, its parameters and workers incentives. The overall result is that the presented social norm based protocol can achieve good performance, the one close to the Pareto efficient outcome.
Posted by Goran Radanovic on Sunday 29 April 2012 at 21:34
Paper:
Y. Singer, and M. Mittal, "Pricing Tasks in Online Labor Markets," in HCOMP11.

In this paper a pricing mechanism is proposed to determine the prices of tasks in online labor markets used for crowdsourcing. In an online labor market, the requesters posts tasks in order to be performed by typically many workers. Usually, the demander wishes to get as many assignments as possible performed with a limited budget. The arrival of the workers is random and hence the price setting is done online. Each worker has a private cost and a maximum number of assignments she can perform. The main contribution of the paper is a pricing mechanism that dynamically determines the prices and allocates assignments to the workers based on their bids. This mechanism is incentive compatible, budget-feasible (i.e., the total price paid does not exceed the budget), and has a near-optimal performance. The mechanism works by maintaining a threshold price which is set to a small value in the beginning and is updated at specific stages based on the observed bids of workers. A worker is allocated some assignments if her bid is less than the threshold price. The approach is evaluated in practice (using Mechanical Turk) and the results show the effectiveness of the mechanism.
Posted by Mehdi Riahi on Monday 30 April 2012 at 18:18