The Research Competition is now closed. Click here to view the winners.

Winning Proposals

Exploring the motivational effects of badges - who do badges appeal to and why?

This project will consider the effectiveness of badges as a motivational tool on two platforms; the Khan Academy (www.khanacademy.org) and the Stack Exchange network (www.stackexchange.com). This project parallels my doctoral research which considers the social impact of social media sites that employ distributed moderation[4]; and the motivations of users to participate in these endeavours [3].

The Khan Academy offers over 2,700 educational videos covering a variety of subjects from basic Arithmetic to Art History; and a programme of exercises through which users can test their knowledge of the subjects offered. Time spent watching videos or completing exercises is rewarded with 'Energy Points', and for reaching specific milestones users are awarded achievement badges. Through the addition of exercises, achievement badges and a voluntary mentoring scheme the Khan Academy has blossomed from a collection of educational videos into a comprehensive and engaging online learning resource.

The Stack Exchange network caters to a different form of learning; generating answers to questions asked by users. The largest websites in the Stack Exchange network (stackoverflow.com, superuser.com and serverfault.com) cater to computer programmers – although more recently the network has expanded to include topics such as Maths and English language usage. On Stackoverflow users ask questions about programming and other users supply answers; users can also vote and comment on each others' answers towards identifying the best answer to the question which was posed.

Stackoverflow's approach to knowledge sharing works well – questions typically receive at least one good answer within 20 minutes of submission [2]. Tagging and community editing are also deployed to facilitate making the archive of answered questions easily navigable. Stackoverflow has now established itself as an important resource for many computer programmers.

The Stack Exchange network awards badges to users for meeting certain targets (e.g. providing ten positively-rated answers). These are accompanied by a reputation score based on the voting response to a user's contributions. One salient difference between the badging systems of the Khan Academy and Stack Exchange is that on Stack Exchange websites a user's reputation score and number of badges are displayed prominently alongside each contribution that user makes. On the Khan Academy website users have limited opportunity to interact with each other and therefore users' energy points and achievement badges are visible only to themselves. Motivations related to reputation and status are thought to be important in the effectiveness of badging systems [1]; and without a window through which users can display their badges these motivations are unlikely to come into play for the Khan Academy.

The principal approach of this project will be to analyse behavioural data recorded by the websites in question. A full history of user behaviour has been obtained for the Stack Exchange network and discussions are ongoing with the Khan Academy regarding access to their data. The following research questions will be addressed through the application of modern statistical techniques such as latent class analysis, cluster analysis for massive data sets and bilinear modelling to these data-sets.

Does the provision of badges motivate users to engage in the rewarded behaviours, and to what degree? Both the Khan Academy and Stack Exchange operated for some time without badges, to address this question I will consider patterns of user behaviour before and after the introduction of badges. There is evidence that one of the badges on Stackoverflow does influence user behaviour [6].

Does the motivational effect of badges apply to all users equally? Montola [5] suggests that while some users find achievement systems interesting and engaging others do not appreciate or understand them. Clustering algorithms and latent class analysis will be applied to data on users and badges in an effort to determine whether there are distinct types of user with regard to badge-seeking behaviour. Are there users who engage in certain behaviours for the purpose of earning a particular badge? Are there badges, the awarding of which, indicates that a user is highly motivated by badge-seeking?

The answers to these questions will have differing implications for Stack Exchange and the Khan Academy. If we consider Microsoft's Xbox Achievements, it is clear that a small sub-set of users are very highly motivated by earning achievements. Indeed whole communities of practice have sprung up around Xbox achievement hunting [7, 8].

This level of dedication to earning badges among a small sub-set of users may have value for the Stack Exchange network; where badges reward sought-after behaviours and a user who provides 1,000 good answers is worth 1,000 users who provide one good answer each. For the Khan Academy however having a small sub-set of users becoming 'compulsive learners' may be counter-productive. One of the principal aims of this project will be to determine why some users but not others become highly motivated to seek badges.

In relation to this it is also important to consider the importance of scores – many systems which offer badges or achievements (including the three discussed here) also award points which serve as a unitary measure of a user's success.

Following initial analysis of behavioural data from the Khan Academy and Stack Exchange; interviews and surveys with the users of these websites will be conducted. Questions will concern the users' demographic information and their motivations to participate on the websites in question. The data generated by interviews and surveys will compliment the behavioural analyses. This data will also be used to address questions about the relative importance of scores and badges, and about the importance of having one's achievements or points displayed for other users to see. A small-scale web-based experiment may also be devised to verify the validity of findings from these surveys. Full ethical approval for these studies will be obtained from a commercial IRB service.

It is expected that the findings which emerge from this project will be of the utmost utility for those involved in creating effective badging systems for on-line learning. As such, I expect to publish at least one article in a peer-reviewed journal and I will be presenting my findings at both US and UK conferences.

Citations

[1] Antin, J. & Churchill, E. (2011), Badges in Social Media: A Social Psychological Perspective, In proceedings of CHI 2011

[2] Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G. & Hartmannm, B. (2011), Design Lessons from the Fastest Q&A Site in the West, In Proceedings of CHI 2011

[3] Mills, R. A. (submitted), Who produces reddit? Diversity in the patterns of behaviour exhibited by users of a Social News website, Information, Communication & Society

[4] Mills, R. A. (2011), Researching Social News – Is reddit.com a mouthpiece for the 'Hive Mind' or a Collective Intelligence approach to Information Overload?, In Proceedings of ETHICOMP 2011 – The Social Impact of Social Computing

[5] Montola, M., Nummenmaa, T., Lucero, A., Boberg, M. & Korhonen, H. (2009), Applying Game Achievement Systems to Enhance User Experience in a Photo Sharing Service, In Proceedings of Mindtrek 2009

[6] Oktay, H., Taylor, B.J. & Jensen, D.D. (2010), Causal Discovery in Social Media Using Quasi-Experimental Designs. In Proceedings of the ACM/SIGKDD Workshop on Social Media Analytics

[7] http://www.xbox360achievements.org

[8] http://www.trueachievements.com