Human Trust in AI: A Review of Empirical Research

In this study, researchers explored the performance of annotators in a user study with 100 college-educated participants. The study aimed to evaluate the accuracy of human annotation in AI-powered text analysis and identify factors that influence annotator performance. The results showed that the median time taken for the study was 35 minutes, and the average payment for each annotation was £4.00. The researchers also filtered out user results with an accuracy below 0.60 during the annotation phase.
To demystify complex concepts, let’s compare annotator performance to cooking a meal. Just as a chef needs to follow a recipe and use their skills to prepare a dish, an annotator needs to follow guidelines and use their knowledge to label text accurately. However, just as different chefs may produce different dishes using the same recipe, annotators can vary in their performance, depending on their expertise and attention to detail.
The study also highlighted the importance of sample size in user studies. While a sample size of 100 is commonly applicable in Human-Computer Interaction literature, it’s essential to consider that many studies involve even fewer participants. For example, some studies have used as few as 8 or 33 participants.
To further simplify the concept, imagine you’re part of a focus group discussing your opinions on a particular topic. Just as different people may have varying levels of expertise and perspectives, annotators can bring unique insights and errors to the table. By understanding these factors and using them to improve AI-powered text analysis, we can create more accurate and reliable systems that leverage human knowledge and expertise.
In conclusion, this study demonstrated that while there are factors that influence annotator performance in AI-powered text analysis, a sample size of 100 is a common standard in user studies. By understanding these factors and using them to improve AI-powered text analysis, we can create more accurate and reliable systems that leverage human knowledge and expertise.