Taking stock of emergency remote course conversions

Watch This Session

Brief Abstract

This session will describe our multipronged online course evaluation approach that includes curriculum mapping, learning experience coding, course workload calculations, multimedia asset review, learning analytics insights and student evaluations to make evidence-based recommendations and suggest a framework for decision-making during the revision cycle.  

Presenters

Additional Authors

Extended Abstract

The emergency remote teaching experience drastically changed the entry point for the online conversion of our core courses. We have been able to begin this cycle of online course design and development with faculty who had a semester of teaching the course online under their belts, data about student behaviors, satisfaction and performance in online versions of the courses and a bevy of multimedia assets created for the course. Our challenge: how to wrangle this pile of information and resources to make the most of these bonuses with a thorough inventory and review? 

We are using a multipronged approach that includes completing online course evaluation rubrics informed by instructional design analysis through curriculum mapping, learning experience coding, course workload calculations, multimedia asset review, learning analytics insights and student evaluations. With a focus on student and faculty experience, we are making evidence-based recommendations and suggesting a framework for decision-making during the revision cycle.  

Context 

Just prior to the shift to emergency remote teaching, we proposed to offer an Advanced MPH Online that would allow us to: 

  • Address gaps in meeting the needs of current on-campus students 
  • Reach new students 
  • Develop more capacity to diversify our educational programs 

The MPH program includes an integrated MPH Core curriculum focusing on foundational knowledge in the field of public health. All MPH students begin the program with a required semester-long introduction to core knowledge in public health, spanning the historical, ethical, biological, social, health systems, environmental, and research methodological foundations of population health. The 15- credit core curriculum is concentrated into one semester in a strategically integrated, 6-course, 18-concentration (educational unit) experience.  

In practice, 18 concentrations translates into over 18 teaching faculty needed for multiple cohorts to accommodate an average of 400 new MPH students each Fall. 

https://ajph.aphapublications.org/doi/10.2105/AJPH.2013.301399 

Opportunity afforded by capacity building for remote teaching and shift to online learning  

All of the Columbia School of Public Health’s classes moved online in mid-March of 2020 to foster social distancing and the majority of courses remain online to this day. “Even in the middle of a pandemic—especially in the middle of a pandemic—public health education continues. The value of public health science and training has never been more evident,” says Dean Linda P. Fried. “We are committed to delivering the same very high quality of education, whether online or in-person, education that is community-based and engaged. I’m so proud of everyone who made this transition happen under these difficult circumstances, and our success in this is a statement about our mission of social solidarity. It is inspiring to see all the ways we can adapt and carry on as a community in the face of a public health emergency.” 

The rapid move to virtual classrooms took a massive, coordinated, behind-the-scenes effort. Faculty and Teaching Assistants mastered the finer points of using Zoom as a classroom, including the use of break-out rooms for small group discussions. The Office of Education took calls, fielded emails and real-time chat messages, and posted an online repository of resources for online pedagogy and tool tips. HR and IT staff provided crucial remote support to ensure a smooth learning experience. 

From the beginning of this effort, we worked to channel all of this effort into capacity building for our longer term online learning goals. We communicated with faculty early on about the differences between emergency remote teaching and well-designed and developed online learning experiences and worked alongside them to accomplish the best possible as time allowed. When the rush to create and deliver the core curriculum fully online in Fall 2020 was over, we immediately turned to the task of thoroughly reviewing what had been accomplished in order to ready the core curriculum as courses for a long term fully online MPH program. 

Purpose and goals 

In order to customize this program for working professionals who were choosing to learn online, we set seven priority goals for student-centered learning experiences: 

  • Manageable workload
  • Regular access to faculty
  • Attention to public health strategic skills
  • Integrated professional context
  • Active learning experiences
  • Frequent peer interaction
  • Authentic assessments 

Because online learning programs are new to the school, we are also establishing our roles and processes as an online learning unit and demonstrating our value to faculty. We set the following goals for faculty experience: 

Design and develop as a team  

  • Collaborate across full-cycle design and development of Adv MPH Online courses 
  • Reduce faculty workload  

Design for reusability and flexibility  

  • Build in efficiencies  
  • Create customizable components  

Design for continuous improvement  

  • Embed outcomes tracking into learning management system 
  • Develop learning analytics dashboard  
  • Systematize revision and release cycles 

Iterative process to establish framework and methodology for review of remote teaching versions of courses 

Starting with the first course in our core curriculum with our priority goals for student and faculty experience in mind, we iteratively established the frameworks, methodologies and instruments for review of the remote teaching versions of all of our core courses. Our multipronged approach includes completing an online course evaluation rubric informed by instructional design analysis through curriculum mapping, taxonomic coding, course workload calculations, and multimedia asset review cross-referenced with learning analytics and student evaluations- informed by the work of our colleagues in higher education across the globe: 

Curriculum mapping 

We began by mapping the learning objectives for every course and concentration to the accrediting bodies requirements for foundational knowledge areas and degree competencies. Then used this mapping schema to provide visual and textual summaries of the types and amounts of course work devoted to each.  

Learning experience coding 

We then coded the mapped learning objectives according to Fink’s taxonomy and the course work activities mapped to those objectives as active or passive leaning to provide a proportional snapshot of the student learning experience.  

Course workload calculations 

All course work activities were calculated using the Course Workload Estimator to indicate how much time is devoted to particular learning objectives and to estimate student workload.  

Multimedia asset review 

In order to help create a framework for the audit of multimedia assets for schoolwide online courses as well as begin work on converting core curriculum courses for the new online MPH program, we developed an information tracking and organization process to include recorded video and audio, slides and graphics and interactive multimedia. We did not include recordings of live class sessions in this process. In addition to using the metadata structure recommended by the Dublin Core Metadata Initiative, we included attention to estimated development time, skills required, learning and engagement, asset reusability and extensibility, production quality, accessibility and diversity and equity. 

Learning analytics 

To glean insight about student behaviors with course work, we used learning analytics form the learning management system and video content management systems used for course and multimedia content delivery. Clicks on content by students were used as proxy for percentage of students engaging with all non-video course materials across cohorts by studio/concentration/unit. For video content, views, duration and heatmap information provided additional information about student behaviors and engagement.  

Student evaluations 

Student course evaluations provide additional information about how students feel about the number, timing and relevancy of course work activities and the rigor and inclusion of strategic skills in the curriculum. Cross-referencing student evaluations with data gathered in this review process helps triangulate findings and shape online course conversion priorities.  

Recommendations and revisions  

To present insights from this comprehensive review to course faculty, we narrowed in on four top recommendations linked to relevant supporting data. For example: 

  • Refine Learning Objectives: See Studio Curriculum Map  
  • Reduce workload: Cross-reference Course Workload Calculations and Learning Analytics 
  • Increase active learning: See suggestions in Online Course Rubric  
  • Revise multimedia assets as needed: See recommendations in Multimedia Asset Review 

Next steps 

We are now working with course faculty leads and concentration teaching faculty to build consensus for course redesign and revisions. It is already apparent that faculty find the reviews enlightening and helpful to focus decision making and that the changes that will occur for the online program will influence changes in the in person core curriculum as well.