Evaluating Effects of Automatic Feedback Aligned to a Learning Progression to Promote Knowledge-In-Use 

This project examines the effect of an assessment system that automatically generates feedback based on students’ open-ended assessment responses in chemistry and physics consistent with a previously-developed learning progression that describes the successively more complex understandings students can develop about electrical interactions. The researchers will design and test an automated assessment scoring system using machine learning. The scoring system will provide individualized feedback to students and class summaries to their teachers. This could then serve as a formative assessment to match an existing high school physical science curriculum designed to meet performance expectations in the Next Generation Science Standards. The project will then examine whether the automatic feedback supports students’ learning outcomes and their development with respect to the learning progression on electrical interactions. The project promotes students’ knowledge of science by engaging them in scientific practices, like modeling, with key disciplinary ideas and using crosscutting concepts to make sense of compelling phenomena and by providing real-time feedback to students. Deepening students’ knowledge of science requires that they have opportunities to solve ill-structured, complex problems and to create models of real-world phenomena. This project serves the national interest by examining how to assess and support students in responding to such problems in science through timely and productive feedback about their performances.

The project addresses two research questions:

(1) What is the effect of automatic feedback on student performance along a previously validated learning progression for physical science aligned with the Next Generation Science Standards?

(2) What is the effect of automatic feedback on how students connect ideas to advance in learning progression levels? 

To address these questions the project uses a curriculum aligned with the Next Generation Science Standards and a validated learning progression for student learning about how electrical interactions allow materials to stick together or be repelled. The project will use an automatic scoring tool that applies natural language processing, image recognition, and supervised machine learning to score students’ explanations and modeling responses. The project embeds the scoring tool in a newly designed web-portal to allow the integration of curriculum materials, assessments, and automatic scoring with real-time feedback. Students respond to formative assessment items embedded in the curriculum; a subset of these are automatically scored by machine learning algorithms. Students receive automatic feedback on their responses based on the computer classification of their responses. Student learning, as defined by advancement on learning progression levels, is measured using item response theory on student performance on the summative assessments given at the start and end of the curriculum units. The project also uses network and cluster analyses for student responses to formative items to identify ways that students respond to feedback and advance along the learning progression. Outcomes from this project will provide guidance on how to provide appropriate feedback to students on science performance assessment items and a machine learning based tool for scoring model representations and explanations.