Evaluation And Workflow Design And Quality Assessment

Although crowdsourcing has been successfully applied in many fields in the past decades, challenges still exist especially in task/workflow design and quality assessment. We take a deeper look at crowdsourcing classification tasks, and explore how task and workflow design can impact the answer quality. Our research is intended to use large knowledge base and citizen science projects as examples and investigate the workflow design considerations and its impact on worker performance as well as overall quality outcome based on statistical, probabilistic, or machine learning models for quality answer prediction, such that optimal workflow design principles can be recommended and applied in other citizen science projects or other human-computer†¦show more content†¦However, challenges still remain no matter it is via volunteered activities (cite{Lease2011,Newman2012}) or paid-microtask platforms (cite{Kittur2013a,Demartini2015,Bernaschina2015}). Some of the most important challen ges include: task and workflow design, and quality assessment. As cite{Kittur2013a} point out, though there are some initial research in complex workflow, we have little knowledge of the broader design space of workflow and it is impossible to simply aggregate multiple independent judgements for complex tasks which may have dependencies between microtasks. Task and workflow design (cite{little2010turkit,Kittur2011,demartini2012zencrowd}) are crucial in ensuring question is properly understood, mitigating the chance of spam and keeping user engaged. They are essential to obtain high quality and quantity of answers. Quality assessment on the fly