Designing, Implementing, and Evaluating Peer Assessment in Massive Open Online Courses: Empirical Research Findings

Brief Abstract

Peer assessment is often used as a strategy in massive open online courses to grade student assignments at scale while personalizing the student experience. Leveraging the findings of a systematic review, this session will discuss key findings related to MOOC peer assessment and propose an agenda for future research.

Extended Abstract

Massive open online courses (MOOCs) have existed for over a decade and, in 2020, enrolled some of their largest numbers yet as students from across the globe flocked to upskilling and reskilling opportunities. Over 180 million students have participated in MOOCs hosted by over 950 institutions and across providers such as Coursera, EdX, and Udacity (Shah, 2020). The scale of MOOCs has challenged the formative assessment strategies used within these platforms and broken the teach-learn-assess cycle that was a fundamental characteristic of pre-MOOC teaching and learning (Suen, 2014). Peer assessment is one such assessment strategy that has been stretched to the extremes in a MOOC environment compared to its pre-MOOC definitions and typologies (e.g., Topping, 1998, 2003).

This session will present the key findings of a systematic review of empirical literature published since 2012 related to peer assessment and feedback in MOOCs. The research team identified 298 potential articles for review, then using a methodical inclusion/exclusion approach, narrowed down the focus to 22 studies for analysis. These studies included details on over 100 MOOCs offered by over 50 unique colleges and universities worldwide. The research team parsed these inclusion studies to answer research questions related to the most common peer assessment structures evident in MOOC research, the most visible scaffolding frameworks, and the methods that are typically used in peer assessment-related research.

As institutions and faculty evaluate their strategy for designing, launching, and evaluating MOOCs, the findings discussed in this presentation will provide attendees with an understanding of common scaffolding practices provided to students within MOOCs to support peer assessment experiences. This session will also discuss common practices for evaluating the peer assessment experience in MOOCs, including surveys, interviews, comparative analyses, and reliability and validity measurements. Opportunities for future research, with a focus on moving past comparative analyses to improvement-focused outcomes will be discussed.

Attendees will be asked to provide information on whether they have been an instructor, instructional designer, or student in a MOOC and whether they have engaged in peer assessment activities in this learning environment. As a presentation in the Graduate Student Discovery Session format, the presenters welcome feedback from attendees on the research topic and analyses used in this systematic review. Attendees will leave the session with a handout of questions that they may consider as they are developing, implementing, and evaluating MOOCs as a faculty member, instructional designer, or institutional leader based on the empirical inclusion studies.

At the end of the session, attendees will be able to describe the most common student scaffolding approaches evident in the empirical literature related to peer assessment in MOOCs. They will also be able to list the most common assessment strategies used in the literature to determine the reliability, validity, or effectiveness of peer assessment in MOOCs. Finally, they will be able to use the handout to identify and facilitate discussions with instructors and instructional designers related to configuring MOOC peer assessment activities at their institution. 

References

Suen, H. K. (2014). Peer assessment for massive open online courses (MOOCs). International Review of Research in Open and Distance Learning, 15(3), 312-327. https://doi.org/10.19173/irrodl.v15i3.1680

Shah, D. (2020, November 30). By the numbers: MOOCs in 2020. The report by Class Central. https://www.classcentral.com/report/mooc-stats-2020/

Topping, K. J. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249–276. https://doi.org/10.3102/00346543068003249

Topping K. J. (2003). Self and peer assessment in school and university: Reliability, validity and utility. In Segers M., Dochy F., Cascallar E. (Eds). Optimising new modes of assessment: In search of qualities and standards (Vol. 1, pp. 55-87). Springer. https://doi.org/10.1007/0-306-48125-1_4