Title:Multi-object Classification via Crowdsourcing with a Reject Option

Abstract:We explore the design of an effective crowdsourcing system for an $M$-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final decision. We consider the novel scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. For example, in mismatched speech transcription, crowd workers who do not know the language may not be able to respond to certain microtasks outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Further, we consider the case where there may be a set of greedy workers in the crowd for whom reward is very important. Greedy workers may respond to a microtask even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy for crowdsourcing with greedy workers, and develop an algorithm to adaptively switch between the two, based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show performance improvement of the proposed approaches compared with conventional majority voting.