Showing 1–7 of 7 results for author: Nissen, J M

Applying Cognitive Diagnostic Models to Mechanics Concept Inventories

Authors: Vy Le, Jayson M. Nissen, Xiuxiu Tang, Yuxiao Zhang, Amirreza Mehrabi, Jason W. Morphew, Hua Hua Chang, Ben Van Dusen

Abstract: In physics education research, instructors and researchers often use research-based assessments (RBAs) to assess students' skills and knowledge. In this paper, we support the development of a mechanics cognitive diagnostic to test and implement effective and equitable pedagogies for physics instruction. Adaptive assessments using cognitive diagnostic models provide significant advantages over fixe… ▽ More In physics education research, instructors and researchers often use research-based assessments (RBAs) to assess students' skills and knowledge. In this paper, we support the development of a mechanics cognitive diagnostic to test and implement effective and equitable pedagogies for physics instruction. Adaptive assessments using cognitive diagnostic models provide significant advantages over fixed-length RBAs commonly used in physics education research. As part of a broader project to develop a cognitive diagnostic assessment for introductory mechanics within an evidence-centered design framework, we identified and tested student models of four skills that cross content areas in introductory physics: apply vectors, conceptual relationships, algebra, and visualizations. We developed the student models in three steps. First, we based the model on learning objectives from instructors. Second, we coded the items on RBAs using the student models. Lastly, we then tested and refined this coding using a common cognitive diagnostic model, the deterministic inputs, noisy 'and' gate (DINA) model. The data included 19,889 students who completed either the Force Concept Inventory, Force and Motion Conceptual Evaluation, or Energy and Momentum Conceptual Survey on the LASSO platform. The results indicated a good to adequate fit for the student models with high accuracies for classifying students with many of the skills. The items from these three RBAs do not cover all of the skills in enough detail, however, they will form a useful initial item bank for the development of the mechanics cognitive diagnostic. △ Less

Submitted 25 January, 2025; v1 submitted 8 March, 2024; originally announced April 2024.

Comments: 13 pages, 3 figures, 11 tables (including appendix). This paper is published by Physical Review Physics Education Research

Online Administration of Research-Based Assessments

Authors: Ben Van Dusen, Mollee Shultz, Jayson M. Nissen, Bethany R. Wilcox, N. G. Holmes, Manher Jariwala, Eleanor W. Close, Steven Pollock

Abstract: The number and use of research-based assessments (RBAs) has grown significantly over the last several decades. Data from RBAs can be compared against national datasets to provide instructors with empirical evidence on the efficacy of their teaching practices. Many physics instructors, however, opt not to use RBAs due to barriers such as having to use class time to administer them. In this article… ▽ More The number and use of research-based assessments (RBAs) has grown significantly over the last several decades. Data from RBAs can be compared against national datasets to provide instructors with empirical evidence on the efficacy of their teaching practices. Many physics instructors, however, opt not to use RBAs due to barriers such as having to use class time to administer them. In this article we examine how these barriers can be mitigated through online administrations of RBAs, particularly through the use of free online RBA platforms that automate administering, scoring, and analyzing RBAs (e.g., the Learning About STEM Student Outcomes [LASSO], Colorado Learning Attitudes About Science Survey for Experimental Physics [E-CLASS], Physics Lab Inventory of Critical thinking [PLIC], and PhysPort DataExplorer platforms). We also explore the research into common concerns of administering RBAs online and conclude with a practical how-to guide for instructors. △ Less

Submitted 7 August, 2020; originally announced August 2020.

Comments: 4 pages, 0 figures

Participation and Performance on Paper- and Computer-Based Low-Stakes Assessments

Authors: Jayson M. Nissen, Manher Jariwala, Eleanor W. Close, Ben Van Dusen

Abstract: High-stakes assessments, such the Graduate Records Examination, have transitioned from paper to computer administration. Low-stakes Research-Based Assessments (RBAs), such as the Force Concept Inventory, have only recently begun this transition to computer administration with online services. These online services can simplify administering, scoring, and interpreting assessments, thereby reducing… ▽ More High-stakes assessments, such the Graduate Records Examination, have transitioned from paper to computer administration. Low-stakes Research-Based Assessments (RBAs), such as the Force Concept Inventory, have only recently begun this transition to computer administration with online services. These online services can simplify administering, scoring, and interpreting assessments, thereby reducing barriers to instructors' use of RBAs. By supporting instructors' objective assessment of the efficacy of their courses, these services can stimulate instructors to transform their courses to improve student outcomes. We investigate the extent to which RBAs administered outside of class with the online Learning About STEM Student Outcomes (LASSO) platform provide equivalent data to tests administered on paper in class, in terms of both student participation and performance. We use an experimental design to investigate the differences between these two assessment conditions with 1,310 students in 25 sections of 3 college physics courses spanning 2 semesters. Analysis conducted using Hierarchical Linear Models indicates that student performance on low-stakes RBAs is equivalent for online (out-of-class) and paper-and-pencil (in-class) administrations. The models also show differences in participation rates across assessment conditions and student grades, but that instructors can achieve participation rates with online assessments equivalent to paper assessments by offering students credit for participating and by providing multiple reminders to complete the assessment. We conclude that online out-of-class administration of RBAs can save class and instructor time while providing participation rates and performance results equivalent to in-class paper-and-pencil tests. △ Less

Submitted 17 November, 2017; originally announced November 2017.

Performance differences for in-class and online administration of low-stakes research-based assessments

Authors: Jayson M. Nissen, Manher Jariwala, Xochith Herrera, Eleanor W. Close, Ben Van Dusen

Abstract: Research-based assessments (RBAs), such as the Force Concept Inventory, have played central roles in many course transformations from traditional lecture-based instruction to research-based teaching methods. In order to support instructors in assessing their courses, the online Learning About STEM Student Outcomes (LASSO) platform simplifies administering, scoring, and interpreting RBAs. Reducing… ▽ More Research-based assessments (RBAs), such as the Force Concept Inventory, have played central roles in many course transformations from traditional lecture-based instruction to research-based teaching methods. In order to support instructors in assessing their courses, the online Learning About STEM Student Outcomes (LASSO) platform simplifies administering, scoring, and interpreting RBAs. Reducing the barriers to using RBAs will support more instructors in objectively assessing the efficacy of their courses and, subsequently, transforming their courses to improve student outcomes. The purpose of this study was to investigate the extent to which RBAs administered online and outside of class with the LASSO platform provided equivalent data to traditional paper and pencil tests administered in class. Research indicates that these two modes of administering assessments provide equivalent data for graded exams that are administered in class. However, little research has focused on ungraded (low-stakes) exams that are administered outside of class. We used an experimental design to investigate the differences between these two test modes. Results indicated that the LASSO platform provided equivalent data to paper and pencil tests. △ Less

Submitted 17 November, 2017; originally announced November 2017.

Systemic inequities in introductory physics courses: the impacts of learning assistants

Authors: Ben Van Dusen, Jayson M. Nissen

Abstract: Creating equitable performance outcomes among students is a focus of many instructors and researchers. One focus of this effort is examining disparities in physics student performance across genders, which is a well-established problem. Another less common focus is disparities across racial and ethnic groups, which may have received less attention due to low representation rates making it difficul… ▽ More Creating equitable performance outcomes among students is a focus of many instructors and researchers. One focus of this effort is examining disparities in physics student performance across genders, which is a well-established problem. Another less common focus is disparities across racial and ethnic groups, which may have received less attention due to low representation rates making it difficult to identify gaps in their performance. In this investigation we examined associations between Learning Assistant (LA) supported courses and improved equity in student performance. We built Hierarchical Linear Models of student performance to investigate how performance differed by gender and by race/ethnicity and how LAs may have moderated those differences. Data for the analysis came from pre-post concept inventories in introductory mechanics courses collected through the Learning About STEM Student Outcomes (LASSO) platform. Our models show that gaps in performance across genders and races/ethnicities were similar in size and increased from pre to post instruction. LA-support is meaningfully and reliably associated with improvement in overall student performance but not with shifts in within-course performance gaps. △ Less

Submitted 15 November, 2017; originally announced November 2017.

Comments: arXiv admin note: text overlap with arXiv:1711.05834

Are inequities in self-efficacy a systemic feature of physics education?

Authors: Jayson M. Nissen

Abstract: There is consistent and growing evidence that physics instruction disproportionately harms female students' self-efficacy, their beliefs about their ability to learn and do physics. This harm is problematic because self-efficacy supports student learning and persistence. Nissen and Shemwell (PhysRevPER, 12, 2016) investigated this harm using an in-the-moment measure of student's self-efficacy stat… ▽ More There is consistent and growing evidence that physics instruction disproportionately harms female students' self-efficacy, their beliefs about their ability to learn and do physics. This harm is problematic because self-efficacy supports student learning and persistence. Nissen and Shemwell (PhysRevPER, 12, 2016) investigated this harm using an in-the-moment measure of student's self-efficacy states, which are dynamic judgments of one's ability to succeed in the activity at hand. Their results indicated that female students experienced much lower self-efficacy states in physics than male students did, and that this gender difference did not occur in other STEM courses. A limitation of their study was that it only investigated a single college physics course. In order to further inform the generalizability of this phenomenon I analyzed a large data set of 35,464 experiences from 4,816 students at 33 secondary schools collected between 1993 and 1997 that was designed to be representative of high school students' experiences. Results confirmed that there was a large gender difference in self-efficacy states in high school physics courses and not in any other courses. The identification of this phenomenon in two very different settings indicates that physics instruction systemically harms female students' belief in their ability to learn and do physics. △ Less

Submitted 29 December, 2016; originally announced December 2016.

A comparison of Hake's g and Cohen's d for analyzing gains on concept inventories

Authors: Jayson M. Nissen, Robert M. Talbot, Amreen Nasim Thompson, Ben Van Dusen

Abstract: Measuring student learning is a complicated but necessary task for understanding the effectiveness of instruction and issues of equity in college STEM courses. Our investigation focused on the implications on claims about student learning that result from choosing between one of two commonly used methods for analyzing shifts in concept inventories. The methods are: Hake's gain (g), which is the mo… ▽ More Measuring student learning is a complicated but necessary task for understanding the effectiveness of instruction and issues of equity in college STEM courses. Our investigation focused on the implications on claims about student learning that result from choosing between one of two commonly used methods for analyzing shifts in concept inventories. The methods are: Hake's gain (g), which is the most common method used in physics education research and other discipline based education research fields, and Cohen's d, which is broadly used in education research and many other fields. Data for the analyses came from the Learning Assistant Supported Student Outcomes (LASSO) database and included test scores from 4,551 students on physics, chemistry, biology, and math concept inventories from 89 courses at 17 institutions from across the United States. We compared the two methods across all of the concept inventories. The results showed that the two methods led to different inferences about student learning and equity due to g being biased in favor of high pretest populations. Recommendations for the analysis and reporting of findings on student learning data are included. △ Less

Submitted 17 February, 2018; v1 submitted 29 December, 2016; originally announced December 2016.

Journal ref: Phys. Rev. Phys. Educ. Res. 14, 010115 (2018)