Showing 1–45 of 45 results for author: Holmes, N

Comparing large language models for supervised analysis of students' lab notes

Authors: Rebeckah K. Fussell, Megan Flynn, Anil Damle, Michael F. J. Fox, N. G. Holmes

Abstract: Recent advancements in large language models (LLMs) hold significant promise in improving physics education research that uses machine learning. In this study, we compare the application of various models to perform large-scale analysis of written text grounded in a physics education research classification problem: identifying skills in students' typed lab notes through sentence-level labeling. S… ▽ More Recent advancements in large language models (LLMs) hold significant promise in improving physics education research that uses machine learning. In this study, we compare the application of various models to perform large-scale analysis of written text grounded in a physics education research classification problem: identifying skills in students' typed lab notes through sentence-level labeling. Specifically, we use training data to fine-tune two different LLMs, BERT and LLaMA, and compare the performance of these models to both a traditional bag of words approach and a few-shot LLM (without fine-tuning).} We evaluate the models based on their resource use, performance metrics, and research outcomes when identifying skills in lab notes. We find that higher-resource models often, but not necessarily, perform better than lower-resource models. We also find that all models estimate similar trends in research outcomes, although the absolute values of the estimated measurements are not always within uncertainties of each other. We use the results to discuss relevant considerations for education researchers seeking to select a model type to use as a classifier. △ Less

Submitted 24 February, 2025; v1 submitted 13 December, 2024; originally announced December 2024.

Determining sensor geometry and gain in a wearable MEG system

Authors: Ryan M. Hill, Gonzalo Reina Rivero, Ashley J. Tyler, Holly Schofield, Cody Doyle, James Osborne, David Bobela, Lukas Rier, Joseph Gibson, Zoe Tanner, Elena Boto, Richard Bowtell, Matthew J. Brookes, Vishal Shah, Niall Holmes

Abstract: Optically pumped magnetometers (OPMs) are compact and lightweight sensors that can measure magnetic fields generated by current flow in neuronal assemblies in the brain. Such sensors enable construction of magnetoencephalography (MEG) instrumentation, with significant advantages over conventional MEG devices including adaptability to head size, enhanced movement tolerance, lower complexity and imp… ▽ More Optically pumped magnetometers (OPMs) are compact and lightweight sensors that can measure magnetic fields generated by current flow in neuronal assemblies in the brain. Such sensors enable construction of magnetoencephalography (MEG) instrumentation, with significant advantages over conventional MEG devices including adaptability to head size, enhanced movement tolerance, lower complexity and improved data quality. However, realising the potential of OPMs depends on our ability to perform system calibration, which means finding sensor locations, orientations, and the relationship between the sensor output and magnetic field (termed sensor gain). Such calibration is complex in OPMMEG since, for example, OPM placement can change from subject to subject (unlike in conventional MEG where sensor locations or orientations are fixed). Here, we present two methods for calibration, both based on generating well-characterised magnetic fields across a sensor array. Our first device (the HALO) is a head mounted system that generates dipole like fields from a set of coils. Our second (the matrix coil (MC)) generates fields using coils embedded in the walls of a magnetically shielded room. Our results show that both methods offer an accurate means to calibrate an OPM array (e.g. sensor locations within 2 mm of the ground truth) and that the calibrations produced by the two methods agree strongly with each other. When applied to data from human MEG experiments, both methods offer improved signal to noise ratio after beamforming suggesting that they give calibration parameters closer to the ground truth than factory settings and presumed physical sensor coordinates and orientations. Both techniques are practical and easy to integrate into real world MEG applications. This advances the field significantly closer to the routine use of OPMs for MEG recording. △ Less

Submitted 11 October, 2024; originally announced October 2024.

Comments: 36 pages, 10 figures

Dynamics of Productive Confirmation Framing in an Introductory Lab

Authors: Ian Descamps, Sophia Jeon, N. G. Holmes, Rachel E. Scherr, David Hammer

Abstract: In introductory physics laboratory instruction, students often expect to confirm or demonstrate textbook physics concepts (Wilcox & Lewandowski, 2017; Hu & Zwickl, 2017; Hu & Zwickl, 2018). This expectation is largely undesirable: labs that emphasize confirmation of textbook physics concepts are unsuccessful at teaching those concepts (Wieman & Holmes, 2015; Holmes et al., 2017) and even in contex… ▽ More In introductory physics laboratory instruction, students often expect to confirm or demonstrate textbook physics concepts (Wilcox & Lewandowski, 2017; Hu & Zwickl, 2017; Hu & Zwickl, 2018). This expectation is largely undesirable: labs that emphasize confirmation of textbook physics concepts are unsuccessful at teaching those concepts (Wieman & Holmes, 2015; Holmes et al., 2017) and even in contexts that don't emphasize confirmation, such expectations can lead to students disregarding or manipulating their data in order to obtain the expected result (Smith et al., 2020). In other words, when students expect their lab activities to confirm a known result, they may relinquish epistemic agency and violate disciplinary practices. We claim that, in other cases, confirmatory expectations can actually support productive disciplinary engagement. In particular, when an expected result is not confirmed, students may enter a productive "troubleshooting" mode (Smith et al., 2020). We analyze the complex dynamics of students' epistemological framing in a lab where student's confirmatory expectations support and even generate epistemic agency and disciplinary practices, including developing original ideas, measures, and apparatuses to apply to the material world. △ Less

Submitted 22 April, 2024; originally announced April 2024.

Bias in physics peer recognition does not explain gaps in perceived recognition

Authors: Meagan Sundstrom, N. G. Holmes

Abstract: Gaining recognition as a physics person by peers is an important contributor to undergraduate students' physics identity and their success in physics courses. Previous research has separately demonstrated that women perceive less recognition from peers than men in their physics courses (perceived recognition) and that women receive fewer nominations from their peers as strong in their physics cour… ▽ More Gaining recognition as a physics person by peers is an important contributor to undergraduate students' physics identity and their success in physics courses. Previous research has separately demonstrated that women perceive less recognition from peers than men in their physics courses (perceived recognition) and that women receive fewer nominations from their peers as strong in their physics course than men (received recognition). The relationship between perceived and received peer recognition for men and women, however, is not well understood. Here we test three plausible models for this relationship. We conduct a large-scale, quantitative study of over 1,700 students enrolled in introductory physics courses at eight institutions in the United States. We directly compare student gender, perceived recognition, and received recognition, controlling for other student demographics and course-level variability. Results show with high precision that, for students receiving the same amount of recognition, and having the same race or ethnicity, academic year, and major, women report significantly lower perceived recognition than men. These findings offer important implications for testable instructional interventions. △ Less

Submitted 30 July, 2024; v1 submitted 30 October, 2023; originally announced October 2023.

What topics of peer interactions correlate with student performance in physics courses?

Authors: L. N. Simpfendoerfer, Meagan Sundstrom, Matthew Dew, N. G. Holmes

Abstract: Research suggests that interacting with more peers about physics course material is correlated with higher student performance. Some studies, however, have demonstrated that different topics of peer interactions may correlate with their performance in different ways, or possibly not at all. In this study, we probe both the peers with whom students interact about their physics course and the partic… ▽ More Research suggests that interacting with more peers about physics course material is correlated with higher student performance. Some studies, however, have demonstrated that different topics of peer interactions may correlate with their performance in different ways, or possibly not at all. In this study, we probe both the peers with whom students interact about their physics course and the particular aspects of the course material about which they interacted in six different introductory physics courses: four lecture courses and two lab courses. Drawing on methods in social network analysis, we replicate prior work demonstrating that, on average, students who interact with more peers in their physics courses have higher final course grades. Expanding on this result, we find that students discuss a wide range of aspects of course material with their peers: concepts, small-group work, assessments, lecture, and homework. We observe that in the lecture courses, interacting with peers about concepts is most strongly correlated with final course grade, with smaller correlations also arising for small-group work and homework. In the lab courses, on the other hand, small-group work is the only interaction topic that significantly correlates with final course grade. We use these findings to discuss how course structures (e.g., grading schemes and weekly course schedules) may shape student interactions and add nuance to prior work by identifying how specific types of student interactions are associated (or not) with performance. △ Less

Submitted 23 October, 2023; originally announced October 2023.

Comments: Submitted to European Journal of Physics

A method to assess trustworthiness of machine coding at scale

Authors: Rebeckah K. Fussell, Emily M. Stump, N. G. Holmes

Abstract: Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader data sets to be analyzed than… ▽ More Physics education researchers are interested in using the tools of machine learning and natural language processing to make quantitative claims from natural language and text data, such as open-ended responses to survey questions. The aspiration is that this form of machine coding may be more efficient and consistent than human coding, allowing much larger and broader data sets to be analyzed than is practical with human coders. Existing work that uses these tools, however, does not investigate norms that allow for trustworthy quantitative claims without full reliance on cross-checking with human coding, which defeats the purpose of using these automated tools. Here we propose a four-part method for making such claims with supervised natural language processing: evaluating a trained model, calculating statistical uncertainty, calculating systematic uncertainty from the trained algorithm, and calculating systematic uncertainty from novel data sources. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes. We also provide a real-world example of using these practices to machine code a data set unseen by human coders. We offer recommendations to guide physics education researchers who may use machine-coding methods in the future. △ Less

Submitted 7 November, 2023; v1 submitted 3 October, 2023; originally announced October 2023.

New perspectives on student reasoning about measurement uncertainty: More or better data

Authors: Andy Schang, Matthew Dew, Emily M. Stump, N. G. Holmes, Gina Passante

Abstract: Uncertainty is an important and fundamental concept in physics education. Students are often first exposed to uncertainty in introductory labs, expand their knowledge across lab courses, and then are introduced to quantum mechanical uncertainty in upper-division courses. This study is part of a larger project evaluating student thinking about uncertainty across these contexts. In this research, we… ▽ More Uncertainty is an important and fundamental concept in physics education. Students are often first exposed to uncertainty in introductory labs, expand their knowledge across lab courses, and then are introduced to quantum mechanical uncertainty in upper-division courses. This study is part of a larger project evaluating student thinking about uncertainty across these contexts. In this research, we investigate advanced physics student thinking about uncertainty by asking them conceptual questions about how a hypothetical distribution of measurements would change if `more' or `better' data were collected in four different experimental scenarios. The scenarios include both classical and quantum experiments, as well as experiments that theoretically result in an expected single value or an expected distribution. This investigation is motivated by our goal of finding insights into students' potential point- and set-like thinking about uncertainty and of shining light on the limitations of those binary paradigms. △ Less

Submitted 29 June, 2023; originally announced June 2023.

Comments: 15 pages, 5 figures, accepted to Physical Review Physics Education Research

Comparing introductory and beyond-introductory students' reasoning about uncertainty

Authors: Emily M. Stump, Mark Hughes, Gina Passante, N. G. Holmes

Abstract: Uncertainty is an important concept in physics laboratory instruction. However, little work has examined how students reason about uncertainty beyond the introductory (intro) level. In this work we aimed to compare intro and beyond-intro students' ideas about uncertainty. We administered a survey to students at 10 different universities with questions probing procedural reasoning about measurement… ▽ More Uncertainty is an important concept in physics laboratory instruction. However, little work has examined how students reason about uncertainty beyond the introductory (intro) level. In this work we aimed to compare intro and beyond-intro students' ideas about uncertainty. We administered a survey to students at 10 different universities with questions probing procedural reasoning about measurement, student-identified sources of uncertainty, and predictive reasoning about data distributions. We found that intro and beyond-intro students answered similarly on questions where intro students already exhibited expert-level reasoning, such as in comparing two data sets with the same mean but different spreads, identifying limitations in an experimental setup, and predicting how a data distribution would change if more data were collected. For other questions, beyond-intro students generally exhibited more expert-like reasoning than intro students, such as when determining whether two sets of data agree, identifying principles of measurement that contribute to spread, and predicting how a data distribution would change if better data were collected. Neither differences in student populations, lab courses taken, nor research experience were able to fully explain the variability between intro and beyond-intro student responses. These results call for further research to better understand how students' ideas about uncertainty develop beyond the intro level. △ Less

Submitted 24 October, 2023; v1 submitted 26 June, 2023; originally announced June 2023.

Comments: 19 pages, 12 figures

Context affects student thinking about sources of uncertainty in classical and quantum mechanics

Authors: Emily M. Stump, Matthew Dew, Gina Passante, N. G. Holmes

Abstract: Measurement uncertainty is an important topic in the undergraduate laboratory curriculum. Previous research on student thinking about experimental measurement uncertainty has focused primarily on introductory-level students' procedural reasoning about data collection and interpretation. In this paper, we extended this prior work to study upper-level students' thinking about sources of measurement… ▽ More Measurement uncertainty is an important topic in the undergraduate laboratory curriculum. Previous research on student thinking about experimental measurement uncertainty has focused primarily on introductory-level students' procedural reasoning about data collection and interpretation. In this paper, we extended this prior work to study upper-level students' thinking about sources of measurement uncertainty across experimental contexts, with a particular focus on classical and quantum mechanics contexts. We developed a survey to probe students' thinking in the generic question "What comes to mind when you think about measurement uncertainty in [classical/quantum] mechanics?" as well as in a range of specific experimental scenarios and interpreted student responses through the lens of availability and accessibility of knowledge pieces. We found that limitations of the experimental setup were most accessible to students in classical mechanics while principles of the underlying physics theory were most accessible to students in quantum mechanics, even in a context in which this theory was not relevant. We recommend that future research probe which sources of uncertainty experts believe are relevant in which contexts and how instruction in both classical and quantum contexts can help students draw on appropriate sources of uncertainty in classical and quantum experiments. △ Less

Submitted 24 October, 2023; v1 submitted 26 June, 2023; originally announced June 2023.

Comments: 15 pages, 8 figures

Who and what gets recognized in peer recognition

Authors: Meagan Sundstrom, L. N. Simpfendoerfer, Annie Tan, Ashley B. Heim, N. G. Holmes

Abstract: Previous work has identified that recognition from others is an important predictor of students' participation, persistence, and career intentions in physics. However, research has also found a gender bias in peer recognition in which student nominations of strong peers in their physics course disproportionately favor men over women. In this study, we draw on methods from social network analysis a… ▽ More Previous work has identified that recognition from others is an important predictor of students' participation, persistence, and career intentions in physics. However, research has also found a gender bias in peer recognition in which student nominations of strong peers in their physics course disproportionately favor men over women. In this study, we draw on methods from social network analysis and find a consistent gender bias in which men disproportionately under-nominate women as strong in their physics course in two offerings of both a lecture course (for science and engineering, but not physics, majors) and a distinct lab course (for science, engineering, and physics majors). We also find in one offering of the lecture course that women disproportionately under-nominate men, contrary to what previous research would predict. We expand on prior work by also probing two data sources related to who and what gets recognized in peer recognition: students' interactions with their peers (who gets recognized) and students' written explanations of their nominations of strong peers (what gets recognized). Results suggest that the nature of the observed gender bias in peer recognition varies between the instructional contexts of lecture and lab. In the lecture course, the gender bias is related to who gets recognized: both men and women disproportionately over-nominate their interaction ties to students of their same gender as strong in the course. In the lab course, the gender bias is also related to what gets recognized: men nominate men more than women because of skills related to interactions, such as being helpful. These findings illuminate the different ways in which students form perceptions of their peers and add nuance to our understanding of the nature of gender bias in peer recognition. △ Less

Submitted 19 January, 2024; v1 submitted 3 May, 2023; originally announced May 2023.

Comments: Submitted to Physical Review Physics Education Research

Designing optimal loop, saddle, and ellipse-based magnetic coils by spherical harmonic mapping

Authors: Peter James Hobson, Noah Louis Hardwicke, Alister Davis, Thomas Smith, Chris Morley, Michael Packer, Niall Holmes, Max Alain Weil, Matthew Brookes, Richard Bowtell, Mark Fromhold

Abstract: Adaptable, low-cost, coils designed by carefully selecting the arrangements and geometries of simple primitive units are used to generate magnetic fields for diverse applications. These extend from magnetic resonance and fundamental physics experiments to active shielding of quantum devices including magnetometers, interferometers, clocks, and computers. However, finding optimal arrangements and g… ▽ More Adaptable, low-cost, coils designed by carefully selecting the arrangements and geometries of simple primitive units are used to generate magnetic fields for diverse applications. These extend from magnetic resonance and fundamental physics experiments to active shielding of quantum devices including magnetometers, interferometers, clocks, and computers. However, finding optimal arrangements and geometries of multiple primitive structures is time-intensive and it is challenging to account for additional constraints, e.g. optical access, during the design process. Here, we demonstrate a general method to find these optimal arrangements. We encode specific symmetries into sets of loops, saddles, and cylindrical ellipses and then solve exactly for the magnetic field harmonics generated by each set. By combining these analytic solutions using computer algebra, we can use numerical techniques to efficiently map the landscape of parameters and geometries which the coils must satisfy. Sets of solutions may be found which generate desired target fields accurately while accounting for complexity and size restrictions. We demonstrate this approach by employing simple configurations of loops, saddles, and cylindrical ellipses to design target linear field gradients and compare their performance with designs obtained using conventional methods. A case study is presented where three optimized arrangements of loops, designed to generate a uniform axial field, a linear axial field gradient, and a quadratic axial field gradient, respectively, are hand-wound around a low-cost, 3D-printed coil former. These coils are used to null the background in a typical laboratory environment, reducing the magnitude of the axial field along the central half of the former's axis from $\left(7.8\pm0.3\right)$ $μ$T (mean $\pm$ st. dev.) to $\left(0.11\pm0.04\right)$ $μ$T. △ Less

Submitted 14 July, 2023; v1 submitted 30 April, 2023; originally announced May 2023.

Comments: 15 pages, 14 figures, 2 tables

Journal ref: IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1-15, 2023, Art no. 1005815

Instructing nontraditional physics labs: Toward responsiveness to student epistemic framing

Authors: Meagan Sundstrom, Rebeckah K. Fussell, Anna McLean Phillips, Mark Akubo, Scott E. Allen, David Hammer, Rachel E. Scherr, N. G. Holmes

Abstract: Research on nontraditional laboratory (lab) activities in physics shows that students often expect to verify predetermined results, as takes place in traditional activities. This understanding of what is taking place, or epistemic framing, may impact their behaviors in the lab, either productively or unproductively. In this paper, we present an analysis of student epistemic framing in a nontraditi… ▽ More Research on nontraditional laboratory (lab) activities in physics shows that students often expect to verify predetermined results, as takes place in traditional activities. This understanding of what is taking place, or epistemic framing, may impact their behaviors in the lab, either productively or unproductively. In this paper, we present an analysis of student epistemic framing in a nontraditional lab to understand how instructional context, specifically instructor behaviors, may shape student framing. We present video data from a lab section taught by an experienced teaching assistant (TA), with 19 students working in seven groups. We argue that student framing in this lab is evidenced by whether or not students articulate experimental predictions and by the extent to which they take up opportunities to construct knowledge (epistemic agency). We show that the TA's attempts to shift student frames generally succeed with respect to experimental predictions but are less successful with respect to epistemic agency. In part, we suggest, the success of the TA's attempts reflects whether and how they are responsive to students' current framing. This work offers evidence that instructors can shift students' frames in nontraditional labs, while also illuminating the complexities of both student framing and the role of the instructor in shifting that framing in this context. △ Less

Submitted 28 April, 2023; originally announced April 2023.

Comments: Submitted to Physical Review Physics Education Research

Understanding interaction network formation across instructional contexts in remote physics courses

Authors: Meagan Sundstrom, Andy Schang, Ashley B. Heim, N. G. Holmes

Abstract: Engaging in interactions with peers is important for student learning. Many studies have quantified patterns of student interactions in in-person physics courses using social network analysis, finding different network structures between instructional contexts (lecture and lab) and styles (active and traditional). Such studies also find inconsistent results as to whether and how student-level vari… ▽ More Engaging in interactions with peers is important for student learning. Many studies have quantified patterns of student interactions in in-person physics courses using social network analysis, finding different network structures between instructional contexts (lecture and lab) and styles (active and traditional). Such studies also find inconsistent results as to whether and how student-level variables (e.g., grades and demographics) relate to the formation of interaction networks. In this cross-sectional research study, we investigate these relationships further by examining lecture and lab interaction networks in four different remote physics courses spanning various instructional styles and student populations. We apply statistical methods from social network analysis -- exponential random graph models -- to measure the relationship between network formation and multiple variables: students' discussion and lab section enrollment, final course grades, gender, and race/ethnicity. Similar to previous studies of in-person courses, we find that remote lecture interaction networks contain large clusters connecting many students, while remote lab interaction networks contain smaller clusters of a few students. Our statistical analysis suggests that these distinct network structures arise from a combination of both instruction-level and student-level variables, including the learning goals of each instructional context, whether assignments are completed in groups or individually, and the distribution of gender and major of students enrolled in a course. We further discuss how these and other variables help to understand the formation of interaction networks in both remote and in-person physics courses. △ Less

Submitted 1 December, 2022; originally announced December 2022.

Comments: To be published in Physical Review Physics Education Research

Introductory physics students' recognition of strong peers: Gender and racial/ethnic bias differ by course level and context

Authors: Meagan Sundstrom, Ashley B. Heim, Barum Park, N. G. Holmes

Abstract: Researchers have pinpointed recognition from others as one of the most important dimensions of students' science and engineering identity. Studies, however, have found gender biases in students' recognition of their peers, with inconsistent patterns across introductory science and engineering courses. Toward finding the source of this variation, we examine whether a gender bias exists in students'… ▽ More Researchers have pinpointed recognition from others as one of the most important dimensions of students' science and engineering identity. Studies, however, have found gender biases in students' recognition of their peers, with inconsistent patterns across introductory science and engineering courses. Toward finding the source of this variation, we examine whether a gender bias exists in students' nominations of strong peers across three different remote, introductory physics courses with varying student populations (varying demographics, majors, and course levels). We also uniquely evaluate possible racial/ethnic biases and probe the relationship between instructional context (whether lecture or laboratory) and recognition. Some of our results replicate previous findings (such as the the association of course grade and small class section enrollment with nominations), while others offer contradictions. Comparing across our three courses and the prior work, results suggest that course level (whether first-year students or beyond-first-year students) might be more associated with a gender bias in peer recognition than other variables. Surprisingly, we also find instances of racial/ethnic biases in favor of students from backgrounds historically underrepresented in science. Finally, we find that the nomination patterns differ when students nominate individuals strong in the lecture material versus laboratory material. This work serves as an important step in determining which courses and contexts exhibit biases in peer recognition, as well as how students' perceptions of one another form in remote teaching environments. △ Less

Submitted 1 December, 2022; originally announced December 2022.

Comments: To be published in Physical Review Physics Education Research

Benchtop magnetic shielding for benchmarking atomic magnetometers

Authors: P. J. Hobson, N. Holmes, P. Patel, J. Chalmers, B. Styles, C. Morley, A. Davis, M. Packer, T. X. Smith, S. Raudonyte, D. Holmes, R. Harrison, D. Woolger, D. Sims, M. J. Brookes, R. W. Bowtell, T. M. Fromhold

Abstract: Here, a benchtop hybrid magnetic shield containing four mumetal cylinders and nine internal flexible printed circuit boards is designed, constructed, tested, and operated. The shield is designed specifically as a test-bed for building and operating ultra-sensitive quantum magnetometers. The geometry and spacing of the mumetal cylinders are optimized to maximize shielding efficiency while maintaini… ▽ More Here, a benchtop hybrid magnetic shield containing four mumetal cylinders and nine internal flexible printed circuit boards is designed, constructed, tested, and operated. The shield is designed specifically as a test-bed for building and operating ultra-sensitive quantum magnetometers. The geometry and spacing of the mumetal cylinders are optimized to maximize shielding efficiency while maintaining Johnson noise $<15$ fT/$\sqrt{}$Hz. Experimental measurements at the shield's center show passive shielding efficiency of $\left(1.0\pm0.1\right){\times}10^6$ for a $0.2$ Hz oscillating field applied along the shield's axis. The nine flexible printed circuit boards generate three uniform fields, which all deviate from perfect uniformity by ${\leq}0.5$% along $50$% of the inner shield axis, and five linear field gradients and one second-order gradient, which all deviate by ${\leq}4$% from perfect linearity and curvature, respectively, over measured target regions. Together, the target field amplitudes are adjusted to minimize the remnant static field along $40$% of the inner shield axis, as mapped using an atomic magnetometer. In this region, the active null reduces the norm of the magnitudes of the three uniform fields and six gradients by factors of $19.5$ and $19.8$, respectively, thereby reducing the total static field from $1.68$ nT to $0.23$ nT. △ Less

Submitted 24 March, 2023; v1 submitted 27 October, 2022; originally announced October 2022.

Comments: 8 pages, 9 figures; This work has been submitted to the IEEE for possible publication

Bespoke magnetic field design for a magnetically shielded cold atom interferometer

Authors: P. J. Hobson, J. Vovrosh, B. Stray, M. Packer, J. Winch, N. Holmes, F. Hayati, K. McGovern, R. Bowtell, M. J. Brookes, K. Bongs, T. M. Fromhold, M. Holynski

Abstract: Quantum sensors based on cold atoms are being developed which produce measurements of unprecedented accuracy. Due to shifts in atomic energy levels, quantum sensors often have stringent requirements on their internal magnetic field environment. Typically, background magnetic fields are attenuated using high permeability magnetic shielding, with the cancelling of residual and introduction of quanti… ▽ More Quantum sensors based on cold atoms are being developed which produce measurements of unprecedented accuracy. Due to shifts in atomic energy levels, quantum sensors often have stringent requirements on their internal magnetic field environment. Typically, background magnetic fields are attenuated using high permeability magnetic shielding, with the cancelling of residual and introduction of quantisation fields implemented with coils inside the shield. The high permeability shield, however, distorts all magnetic fields, including those generated inside the sensor. Here, we demonstrate a solution by designing multiple coils overlaid on a 3D-printed former to generate three uniform and three constant linear gradient magnetic fields inside the capped cylindrical magnetic shield of a cold atom interferometer. The fields are characterised in-situ and match their desired forms to high accuracy. For example, the uniform transverse field, $B_x$, deviates by less than $0.2$% over more than $40$% of the length of the shield. We also map the field directly using the cold atoms and investigate the potential of the coil system to reduce bias from the quadratic Zeeman effect. This coil design technology enables targeted field compensation over large spatial volumes and has the potential to reduce systematic shifts and noise in numerous cold atom systems. △ Less

Submitted 23 June, 2022; v1 submitted 9 October, 2021; originally announced October 2021.

Comments: 12 pages, 9 figures

Journal ref: Sci. Rep. 12, 10520 (2022)

Magnetic Field Design in a Cylindrical High-Permeability Shield: The Combination of Simple Building Blocks and a Genetic Algorithm

Authors: M. Packer, P. J. Hobson, A. Davis, N. Holmes, J. Leggett, P. Glover, N. L. Hardwicke, M. J. Brookes, R. Bowtell, T. M. Fromhold

Abstract: Magnetically-sensitive experiments and newly-developed quantum technologies with integrated high-permeability magnetic shields require increasing control of their magnetic field environment and reductions in size, weight, power and cost. However, magnetic fields generated by active components are distorted by high-permeability magnetic shielding, particularly when they are close to the shield's su… ▽ More Magnetically-sensitive experiments and newly-developed quantum technologies with integrated high-permeability magnetic shields require increasing control of their magnetic field environment and reductions in size, weight, power and cost. However, magnetic fields generated by active components are distorted by high-permeability magnetic shielding, particularly when they are close to the shield's surface. Here, we present an efficient design methodology for creating desired static magnetic field profiles by using discrete coils electromagnetically-coupled to a cylindrical passive magnetic shield. We utilize a modified Green's function solution that accounts for the interior boundary conditions on a closed finite-length high-permeability cylindrical magnetic shield, and determine simplified expressions when a cylindrical coil approaches the interior surface of the shield. We use an analytic formulation of simple discrete building blocks to provide a complete discrete coil basis to generate any physically-attainable magnetic field inside the shield. We then use a genetic algorithm to find optimized discrete coil structures composed of this basis. We use our methodology to generate an improved linear axial gradient field, $\mathrm{d}B_z/\mathrm{d}z$, and transverse bias field, $B_x$. These optimized structures increase, by a factor of seven and three compared to the standard configurations, the volume in which the desired and achieved fields agree within $1\%$ accuracy, respectively. This coil design method can be used to optimize active--passive magnetic field shaping systems that are compact and simple to manufacture, enabling accurate magnetic field control in spatially-confined experiments at low cost. △ Less

Submitted 4 March, 2022; v1 submitted 7 July, 2021; originally announced July 2021.

Comments: The authors M. Packer and P. J. Hobson have contributed equally to this work. 24 pages, 16 figures

Journal ref: J. Appl. Phys. 131, 093902 (2022)

Not engaging with problems in the lab: Students' navigation of conflicting data and models

Authors: Anna McLean Phillips, Meagan Sundstrom, David G. Wu, N. G. Holmes

Abstract: With the adoption of instructional laboratories (labs) that require students to make their own decisions, there is a need to better understand students' activities as they make sense of their data and decide how to proceed. In particular, understanding when students do not engage productively with unexpected data may provide insights into how to better support students in more open-ended labs. We… ▽ More With the adoption of instructional laboratories (labs) that require students to make their own decisions, there is a need to better understand students' activities as they make sense of their data and decide how to proceed. In particular, understanding when students do not engage productively with unexpected data may provide insights into how to better support students in more open-ended labs. We examine video and audio data from groups within a lab session where students were expected to find data inconsistent with the predictions of two models. In prior work, we examined the actions of the four groups that productively grapple with this designed problem. Here, we analyze the engagement of the three groups that do not. We conducted three phases of analysis: 1) documenting large scale behaviors and time spent in on-topic discussion, 2) analyzing interactions with the teaching assistant, and 3) identifying students' framing--their expectations for what is taking place--when they were discussing their data. Our Phase 1 and 2 analysis show only minor differences between the groups that engaged with the problem and those that did not. Our Phase 3 analysis demonstrated that the groups that did not engage with the problem framed the lab activity as about confirming a known result or as a series of hoops to jump through to fulfill assignment requirements. Implications for instruction include supporting teaching assistants to attend to students' framing and agency within laboratory classrooms. △ Less

Submitted 2 March, 2021; originally announced March 2021.

Journal ref: Phys. Rev. Phys. Educ. Res. 17, 020112 (2021)

Planar Coil Optimization in a Magnetically Shielded Cylinder

Authors: M. Packer, P. J. Hobson, N. Holmes, J. Leggett, P. Glover, M. J. Brookes, R. Bowtell, T. M. Fromhold

Abstract: Hybrid magnetic shields with both active field generating components and high-permeability magnetic shielding are increasingly needed for a variety of technologies and experiments that require precision-controlled magnetic field environments. However, the fields generated by the active components interact with the passive magnetic shield, distorting the desired field profiles. Consequently, optimi… ▽ More Hybrid magnetic shields with both active field generating components and high-permeability magnetic shielding are increasingly needed for a variety of technologies and experiments that require precision-controlled magnetic field environments. However, the fields generated by the active components interact with the passive magnetic shield, distorting the desired field profiles. Consequently, optimization of the active components needed to generate user-specified target fields must include coupling to the high-permeability passive components. Here, we consider the optimization of planar active systems, on which an arbitrary static current flows, coupled to a closed high-permeability cylindrical shield. We modify the Green's function for the magnetic vector potential to match boundary conditions on the shield's interior surface, enabling us to construct an inverse optimization problem to design planar coils that generate user-specified magnetic fields inside high-permeability shields. We validate our methodology by designing two bi-planar hybrid active--passive systems, which generate a constant transverse field, $\mathbf{B}=\mathbf{\hat{x}}$, and a linear field gradient, $\mathbf{B}=(-x~\mathbf{\hat{x}}-y~\mathbf{\hat{y}}+2z~\mathbf{\hat{z}})$, respectively. For both systems, the inverse-optimized magnetic field profiles agree well with forward numerical simulations. Our design methodology is accurate and flexible, facilitating the miniaturization of high-performance hybrid magnetic field generating technologies with strict design constraints and spatial limitations. △ Less

Submitted 3 June, 2021; v1 submitted 4 January, 2021; originally announced January 2021.

Comments: 24 pages, 10 figures

Journal ref: Phys. Rev. Applied 15, 064006 (2021)

Online Administration of Research-Based Assessments

Authors: Ben Van Dusen, Mollee Shultz, Jayson M. Nissen, Bethany R. Wilcox, N. G. Holmes, Manher Jariwala, Eleanor W. Close, Steven Pollock

Abstract: The number and use of research-based assessments (RBAs) has grown significantly over the last several decades. Data from RBAs can be compared against national datasets to provide instructors with empirical evidence on the efficacy of their teaching practices. Many physics instructors, however, opt not to use RBAs due to barriers such as having to use class time to administer them. In this article… ▽ More The number and use of research-based assessments (RBAs) has grown significantly over the last several decades. Data from RBAs can be compared against national datasets to provide instructors with empirical evidence on the efficacy of their teaching practices. Many physics instructors, however, opt not to use RBAs due to barriers such as having to use class time to administer them. In this article we examine how these barriers can be mitigated through online administrations of RBAs, particularly through the use of free online RBA platforms that automate administering, scoring, and analyzing RBAs (e.g., the Learning About STEM Student Outcomes [LASSO], Colorado Learning Attitudes About Science Survey for Experimental Physics [E-CLASS], Physics Lab Inventory of Critical thinking [PLIC], and PhysPort DataExplorer platforms). We also explore the research into common concerns of administering RBAs online and conclude with a practical how-to guide for instructors. △ Less

Submitted 7 August, 2020; originally announced August 2020.

Comments: 4 pages, 0 figures

Preliminary evidence for available roles in mixed-gender and all-women lab groups

Authors: N. G. Holmes, Z. Yasemin Kalender

Abstract: Group work during lab instruction can be a source of inequity between male and female students. In this preliminary study, we explored the activities male and female students take on during a lab session at a university in Denmark. Different from many studies, the class was majority-female, so three of the seven groups were all female and the rest were mixed-gender. We found that students in mixed… ▽ More Group work during lab instruction can be a source of inequity between male and female students. In this preliminary study, we explored the activities male and female students take on during a lab session at a university in Denmark. Different from many studies, the class was majority-female, so three of the seven groups were all female and the rest were mixed-gender. We found that students in mixed-gender groups divide tasks in similar ways to mixed-gender groups at North American institutions, with men handling the equipment and women handling the computer more often. We also found that women in single-gender groups took on each of the available roles with approximately equal frequency, but women in single-gender groups spent more time on the equipment than students in mixed-gender groups. We interpret the results through poststructual gender theory and the notion of `doing physics' and `doing gender' in physics labs. △ Less

Submitted 29 July, 2020; originally announced July 2020.

Comments: 6 pages, 3 figures

What do equitable physics lab groups look like in light of inchargeness?

Authors: Sophia M. Jeon, Eleanor C. Sayre, N. G. Holmes

Abstract: In physics labs, students experience a wide range of equitable and inequitable interactions. We developed a methodology to characterize different lab groups in terms of their bid exchanges and inchargeness. An equitable group is one in which every student's bids are heard and acknowledged. Our analysis of equitable and inequitable groups raises questions about how inchargeness and gender interact… ▽ More In physics labs, students experience a wide range of equitable and inequitable interactions. We developed a methodology to characterize different lab groups in terms of their bid exchanges and inchargeness. An equitable group is one in which every student's bids are heard and acknowledged. Our analysis of equitable and inequitable groups raises questions about how inchargeness and gender interact to affect the functionality of a lab group. △ Less

Submitted 25 July, 2020; originally announced July 2020.

Comments: Submitted to ICLS 2020

Sense of agency, gender, and students' perception in open-ended physics labs

Authors: Z. Yasemin Kalender, Martin Stein, N. G. Holmes

Abstract: Instructional physics labs are critical junctures for many STEM majors to develop an understanding of experimentation in the sciences. Students can acquire useful experimental skills and grow their identities as scientists. However, many traditionally-instructed labs do not necessarily involve authentic physics experimentation features in their curricula. Recent research calls for a reformation in… ▽ More Instructional physics labs are critical junctures for many STEM majors to develop an understanding of experimentation in the sciences. Students can acquire useful experimental skills and grow their identities as scientists. However, many traditionally-instructed labs do not necessarily involve authentic physics experimentation features in their curricula. Recent research calls for a reformation in undergraduate labs to incorporate more student agency and choice in the learning processes. In our institution, we have adopted open-ended lab teaching in the introductory physics courses. By using reformed curricula that provide higher student agency, we analyzed approximately 100 students in the introductory-level lab courses to examine their views towards the open-ended physics labs. Between the start and the end of the semester, we found a statistically significant shift in students' perceptions about the agency afforded in lab activities. We also examined students' responses to "Which lab unit was your favorite and why?". The analysis showed that majority of the students preferred Project Lab, which had the highest student agency and coding analysis showed that "freedom" was the most frequent response for students' reason for picking Project Lab. Finally, we also examined student views across gender and found no significant gender effect on students' sense of agency. △ Less

Submitted 14 July, 2020; originally announced July 2020.

Student evaluation of more or better experimental data in classical and quantum mechanics

Authors: Courtney L. While, Emily M. Stump, N. G. Holmes, Gina Passante

Abstract: Prior research has shown that physics students often think about experimental procedures and data analysis very differently from experts. One key framework for analyzing student thinking has found that student thinking is more point-like, putting emphasis on the results of a single experimental trial, whereas set-like thinking relies on the results of many trials. Recent work, however, has found t… ▽ More Prior research has shown that physics students often think about experimental procedures and data analysis very differently from experts. One key framework for analyzing student thinking has found that student thinking is more point-like, putting emphasis on the results of a single experimental trial, whereas set-like thinking relies on the results of many trials. Recent work, however, has found that students rarely fall into one of these two extremes, which may be a limitation of how student thinking is evaluated. Measurements of student thinking have focused on probing students' procedural knowledge by asking them, for example, what steps they might take next in an experiment. Two common refrains are to collect more data, or to improve the experiment and collect better data. In both of these cases, the underlying reasons behind student responses could be based in point-like or set-like thinking. In this study we use individual student interviews to investigate how advanced physics students believe the collection of more and better data will affect the results of a classical and a quantum mechanical experiment. The results inform future frameworks and assessments for characterizing students thinking between the extremes of point and set reasoning in both classical and quantum regimes. △ Less

Submitted 14 July, 2020; originally announced July 2020.

Comments: Accepted for publication in 2020 PERC Proceedings

Student reasoning about sources of experimental measurement uncertainty in quantum versus classical mechanics

Authors: Emily M. Stump, Courtney L. White, Gina Passante, N. G. Holmes

Abstract: Measurement uncertainty and experimental error are important concepts taught in undergraduate physics laboratories. Although student ideas about error and uncertainty in introductory classical mechanics lab experiments have been studied extensively, there is relatively limited research on student thinking about experimental measurement uncertainty in quantum mechanics. In this work, we used semi-s… ▽ More Measurement uncertainty and experimental error are important concepts taught in undergraduate physics laboratories. Although student ideas about error and uncertainty in introductory classical mechanics lab experiments have been studied extensively, there is relatively limited research on student thinking about experimental measurement uncertainty in quantum mechanics. In this work, we used semi-structured interviews to study advanced physics students' interpretations of fictitious data distributions from two common undergraduate laboratory experiments in quantum mechanics and one in classical mechanics. To analyze these interpretations, we developed a coding scheme that classifies student responses based on what factors they believe create uncertainty and differentiates between different types of uncertainty (e.g. imprecision, inaccuracy). We found that participants in our study expressed a variety of ideas about measurement uncertainty that varied with the context (classical/quantum) and the type of uncertainty. △ Less

Submitted 18 July, 2020; v1 submitted 13 July, 2020; originally announced July 2020.

Optimal Inverse Design of Magnetic Field Profiles in a Magnetically Shielded Cylinder

Authors: M. Packer, P. J. Hobson, N. Holmes, J. Leggett, P. Glover, M. J. Brookes, R. Bowtell, T. M. Fromhold

Abstract: Magnetic shields that use both active and passive components to enable the generation of a tailored low-field environment are required for many applications in science, engineering, and medical imaging. Until now, accurate field nulling, or field generation, has only been possible over a small fraction of the overall volume of the shield. This is due to the interaction between the active field-gen… ▽ More Magnetic shields that use both active and passive components to enable the generation of a tailored low-field environment are required for many applications in science, engineering, and medical imaging. Until now, accurate field nulling, or field generation, has only been possible over a small fraction of the overall volume of the shield. This is due to the interaction between the active field-generating components and the surrounding high-permeability passive shielding material. In this paper, we formulate the interaction between an arbitrary static current flow on a cylinder and an exterior closed high-permeability cylinder. We modify the Green's function for the magnetic vector potential and match boundary conditions on the shield's interior surface to calculate the total magnetic field generated by the system. We cast this formulation into an inverse optimization problem to design active--passive magnetic field shaping systems that accurately generate any physical static magnetic field in the interior of a closed cylindrical passive shield. We illustrate this method by designing hybrid systems that generate a range of magnetic field profiles to high accuracy over large interior volumes, and simulate them in real-world shields whose passive components have finite permeability, thickness, and axial entry holes. Our optimization procedure can be adapted to design active--passive magnetic field shaping systems that accurately generate any physical user-specified static magnetic field in the interior of a closed cylindrical shield of any length, enabling the development and miniaturization of systems that require accurate magnetic shielding and control. △ Less

Submitted 7 September, 2020; v1 submitted 4 June, 2020; originally announced June 2020.

Comments: 22 pages, 9 figures, 1 table

Journal ref: Phys. Rev. Applied 14, 054004 (2020)

Group roles in unstructured labs show inequitable gender divide

Authors: Katherine N. Quinn, Michelle M. Kelley, Kathryn L. McGill, Emily M. Smith, Zachary Whipps, N. G. Holmes

Abstract: Instructional labs are being transformed to better reflect authentic scientific practice, often by removing aspects of pedagogical structure to support student agency and decision-making. We explored how these changes impact men's and women's participation in group work associated with labs through clustering methods on the quantified behavior of students. We compared the group roles students take… ▽ More Instructional labs are being transformed to better reflect authentic scientific practice, often by removing aspects of pedagogical structure to support student agency and decision-making. We explored how these changes impact men's and women's participation in group work associated with labs through clustering methods on the quantified behavior of students. We compared the group roles students take on in two different types of instructional settings; (1) highly structured traditional labs, and (2) less structured inquiry-based labs. Students working in groups in the inquiry-based (less structured) labs assumed different roles within their groups, however men and women systematically took on different roles and men behaved differently when in single- versus mixed-gender groups. We found no such systematic differences in role division among male and female students in the traditional (highly-structured) labs. Students in the inquiry-based labs were not overtly assigned these roles, indicating that the inequitable division of roles was not a result of explicit assignment. Our results highlight the importance of structuring equitable group dynamics in educational settings, as a gendered division of roles can emerge without active intervention. As the culture in physics evolves to remove systematic gender biases in the field, instructors in educational settings must not only remove explicitly biased aspects of curricula but also take active steps to ensure that potentially discriminatory aspects are not inadvertently reinforced. △ Less

Submitted 15 May, 2020; originally announced May 2020.

Ready Student One: Exploring the predictors of student learning in virtual reality

Authors: J. Madden, S. Pandita, J. P. Schuldt, B. Kim, A. S. Won, N. G. Holmes

Abstract: Immersive virtual reality (VR) has enormous potential for education, but classroom resources are limited. Thus, it is important to identify whether and when VR provides sufficient advantages over other modes of learning to justify its deployment. In a between-subjects experiment, we compared three methods of teaching Moon phases (a hands-on activity, VR, and a desktop simulation) and measured stud… ▽ More Immersive virtual reality (VR) has enormous potential for education, but classroom resources are limited. Thus, it is important to identify whether and when VR provides sufficient advantages over other modes of learning to justify its deployment. In a between-subjects experiment, we compared three methods of teaching Moon phases (a hands-on activity, VR, and a desktop simulation) and measured student improvement on existing learning and attitudinal measures. While a substantial majority of students preferred the VR experience, we found no significant differences in learning between conditions. However, we found differences between conditions based on gender, which was highly correlated with experience with video games. These differences may indicate certain groups have an advantage in the VR setting. △ Less

Submitted 14 May, 2020; v1 submitted 24 October, 2019; originally announced October 2019.

Comments: 28 pages, 7 figures, 4 tables. Published in PLOS ONE March 25, 2020

Journal ref: PLOS ONE 15(3): e0229788 (2020)

Participating in the Physics Lab: Does Gender Matter?

Authors: N. G. Holmes, Ido Roll, D. A. Bonn

Abstract: It is a well-studied notion that women are under-represented in the physical sciences, with a "leaky pipeline" metaphor describing how the number of women decreases at higher levels in academia[1,2]. It is unclear, however, where the major leaks exist and what factors are responsible for this[2]. Our focus here is on women in physics with an emphasis on practical laboratory work. It is a well-studied notion that women are under-represented in the physical sciences, with a "leaky pipeline" metaphor describing how the number of women decreases at higher levels in academia[1,2]. It is unclear, however, where the major leaks exist and what factors are responsible for this[2]. Our focus here is on women in physics with an emphasis on practical laboratory work. △ Less

Submitted 18 April, 2019; originally announced May 2019.

Journal ref: Physics in Canada, 70(2), 84-86 (2014)

Quantifying critical thinking: Development and validation of the Physics Lab Inventory of Critical thinking (PLIC)

Authors: Cole Walsh, Katherine N. Quinn, C. Wieman, N. G. Holmes

Abstract: Introductory physics lab instruction is undergoing a transformation, with increasing emphasis on developing experimentation and critical thinking skills. These changes present a need for standardized assessment instruments to determine the degree to which students develop these skills through instructional labs. In this article, we present the development and validation of the Physics Lab Inventor… ▽ More Introductory physics lab instruction is undergoing a transformation, with increasing emphasis on developing experimentation and critical thinking skills. These changes present a need for standardized assessment instruments to determine the degree to which students develop these skills through instructional labs. In this article, we present the development and validation of the Physics Lab Inventory of Critical thinking (PLIC). We define critical thinking as the ability to use data and evidence to decide what to trust and what to do. The PLIC is a 10-question, closed-response assessment that probes student critical thinking skills in the context of physics experimentation. Using interviews and data from 5584 students at 29 institutions, we demonstrate, through qualitative and quantitative means, the validity and reliability of the instrument at measuring student critical thinking skills. This establishes a valuable new assessment instrument for instructional labs. △ Less

Submitted 21 January, 2019; originally announced January 2019.

Comments: 16 pages, 4 figures

Operationalizing the AAPT Learning Goals for the Lab

Authors: N. G. Holmes, Emily M. Smith

Abstract: Calls for reform to instructional labs means many instructors and departments are facing the daunting task of identifying goals for their introductory lab courses. Fortunately, the American Association of Physics Teachers (AAPT) released a set of recommendations for learning goals for the lab to support lab redevelopment [1]. Here we outline the process we have undergone to identify a set of learn… ▽ More Calls for reform to instructional labs means many instructors and departments are facing the daunting task of identifying goals for their introductory lab courses. Fortunately, the American Association of Physics Teachers (AAPT) released a set of recommendations for learning goals for the lab to support lab redevelopment [1]. Here we outline the process we have undergone to identify a set of learning goals for the labs that operationalize those provided by the AAPT. We also provide two examples of newly developed lab activities that aim to meet those goals to demonstrate this operationalization. We aim to provide departments and instructors with a few ideas of a procedure that they can follow or a set of goals that they can use to align lab instruction with the AAPT learning goals. △ Less

Submitted 31 August, 2018; originally announced September 2018.

Comments: 6 pages, 4 figures, accepted to The Physics Teacher

Assessment of critical thinking in physics labs

Authors: Cole Walsh, Katherine N. Quinn, N. G. Holmes

Abstract: Despite the significant amount of time undergraduate students spend in introductory physics labs, there is little consensus on instructional goals and accepted diagnostic assessments for these labs. In response to these issues, we have developed the Physics Lab Inventory of Critical thinking (PLIC) to assess students' proficiency with critical thinking in a physics lab context. Specifically, the P… ▽ More Despite the significant amount of time undergraduate students spend in introductory physics labs, there is little consensus on instructional goals and accepted diagnostic assessments for these labs. In response to these issues, we have developed the Physics Lab Inventory of Critical thinking (PLIC) to assess students' proficiency with critical thinking in a physics lab context. Specifically, the PLIC aims to evaluate students' skills in making sense of data, variability, models, and experimental methods and to assess the effectiveness of lab courses at developing these skills. Here, we discuss the statistical and expert validation of this instrument using 2681 valid student responses collected from 12 institutions during the 2017-2018 academic year. As a part of our validation, we address the impact of lab design on student performance on the instrument. △ Less

Submitted 30 August, 2018; originally announced August 2018.

Virtual Reality as a Teaching Tool for Moon Phases and Beyond

Authors: J. H. Madden, A. S. Won, J. P. Schuldt, B. Kim, S. Pandita, Y. Sun, T. J. Stone, N. G. Holmes

Abstract: A ball on a stick is a common and simple activity for teaching the phases of the Moon. This activity, like many others in physics and astronomy, gives students a perspective they otherwise could only imagine. For Moon phases, a third person view and control over time allows students to rapidly build a mental model that connects all the moving parts. Computer simulations of many traditional physics… ▽ More A ball on a stick is a common and simple activity for teaching the phases of the Moon. This activity, like many others in physics and astronomy, gives students a perspective they otherwise could only imagine. For Moon phases, a third person view and control over time allows students to rapidly build a mental model that connects all the moving parts. Computer simulations of many traditional physics and astronomy activities provide new features, controls, or vantage points to enhance learning beyond a hands-on activity. Virtual reality provides the capabilities of computer simulations and embodied cognition experiences through a hands-on activity making it a natural step to improve learning. We recreated the traditional ball-and-stick moon phases activity in virtual reality and compared participant learning using this simulation with using traditional methods. We found a strong participant preference for VR relative to the traditional methods. However, we observed no difference across conditions in average levels of performance on a pre/post knowledge test. △ Less

Submitted 12 November, 2018; v1 submitted 30 July, 2018; originally announced July 2018.

Comments: 4 pages - Submitted to the 2018 Physics Education Research Conference

Who does what now? How physics lab instruction impacts student behaviors

Authors: Katherine N. Quinn, Kathryn L. McGill, Michelle M. Kelley, Emily M. Smith, N. G. Holmes

Abstract: While laboratory instruction is a cornerstone of physics education, the impact of student behaviours in labs on retention, persistence in the field, and the formation of students' physics identity remains an open question. In this study, we performed in-lab observations of student actions over two semesters in two pedagogically different sections of the same introductory physics course. We used a… ▽ More While laboratory instruction is a cornerstone of physics education, the impact of student behaviours in labs on retention, persistence in the field, and the formation of students' physics identity remains an open question. In this study, we performed in-lab observations of student actions over two semesters in two pedagogically different sections of the same introductory physics course. We used a cluster analysis to identify different categories of student behaviour and analyzed how they correlate with lab structure and gender. We find that, in lab structures which fostered collaborative group work and promoted decision making, there was a task division along gender lines with respect to laptop and equipment usage (and found no such divide among students in guided verification labs). △ Less

Submitted 25 July, 2018; originally announced July 2018.

Comments: 4 pages, 3 figures, 3 tables

Confirming what we know: Understanding questionable research practices in intro physics labs

Authors: Martin M. Stein, Emily M. Smith, N. G. Holmes

Abstract: Many institutions are changing the focus of their introductory physics labs from verifying physics content towards teaching students about the skills and nature of science. As instruction shifts, so too will the ways students approach and behave in the labs. In this study, we evaluated students' lab notes from an early activity in an experimentation-focused lab course. We found that about 30% of s… ▽ More Many institutions are changing the focus of their introductory physics labs from verifying physics content towards teaching students about the skills and nature of science. As instruction shifts, so too will the ways students approach and behave in the labs. In this study, we evaluated students' lab notes from an early activity in an experimentation-focused lab course. We found that about 30% of student groups (out of 107 groups at three institutions) recorded questionable research practices in their lab notes, such as subjective interpretations of results or manipulating equipment and data. The large majority of these practices were associated with confirmatory goals, which we suspect stem from students' prior exposure to verification labs. We propose ways for experimentation-focused labs to better engage students in the responsible conduct of research and authentic scientific practice. △ Less

Submitted 12 July, 2018; originally announced July 2018.

Comments: 4 pages, 4 figures

Surprise! Shifting students away from model-verifying frames in physics labs

Authors: Emily M. Smith, Martin M. Stein, N. G. Holmes

Abstract: Framing affects how students interpret, approach, and accomplish tasks. Little is known, however, about how students frame tasks in physics labs. During the first lab of a sequence designed to teach students about modeling and critical thinking with data, students test a simple model of a pendulum that breaks down with improved measurements. Using in-lab video and follow-up interviews, we identifi… ▽ More Framing affects how students interpret, approach, and accomplish tasks. Little is known, however, about how students frame tasks in physics labs. During the first lab of a sequence designed to teach students about modeling and critical thinking with data, students test a simple model of a pendulum that breaks down with improved measurements. Using in-lab video and follow-up interviews, we identified students' frequent use of a model-verifying frame that substantially interferes with the instructional goals. We present a case study analysis of two students who approach the lab with a model-verifying frame, engage in problematic behaviors including questionable research practices, but later shift their frames to accommodate goals aligned with instructional intention. As instructors transition their instructional labs to open-inquiry experiences, an activity that directly challenges the model-verifying frame may be productive for shifting students away from this problematic frame to one that supports their engagement in authentic experimentation. △ Less

Submitted 12 July, 2018; originally announced July 2018.

Interview validation of the Physics Lab Inventory of Critical thinking (PLIC)

Authors: Katherine N. Quinn, Carl Wieman, N. G. Holmes

Abstract: Although an important goal of introductory physics labs is to train students in scientific reasoning and critical thinking, currently there are no standard tests in physics designed to assess such skills. We are in the process of developing and validating the Physics Lab Inventory of Critical thinking (PLIC), an assessment to probe students' critical thinking abilities in physics lab courses. The… ▽ More Although an important goal of introductory physics labs is to train students in scientific reasoning and critical thinking, currently there are no standard tests in physics designed to assess such skills. We are in the process of developing and validating the Physics Lab Inventory of Critical thinking (PLIC), an assessment to probe students' critical thinking abilities in physics lab courses. The instrument asks students to critique a set of experimental methods and data and use them to evaluate a particular physical model (the period of a mass on a spring). Currently, we are validating the closed-response survey through interviews with students and present the results of 12 such interviews here. We describe a trend that has emerged from these interviews, with students' reasoning falling into three main patterns of behavior: selecting all options, cuing on keywords, and critically analyzing. We have found ways to shift students to the last and more desirable behavior. We discuss ways in which these findings are likely relevant to the design of other concept inventories. △ Less

Submitted 7 February, 2018; originally announced February 2018.

Comments: 4 pages

Seeing the real world: Comparing learning from verification labs and traditional or enhanced lecture demonstrations

Authors: Emily M. Smith, N. G. Holmes

Abstract: Instructors in introductory physics courses often use labs and demonstrations to reinforce that the physics equations introduced in lectures and textbooks describe what actually happens in the real world. The surface features and instructional learning goals of such labs and demonstrations may seem quite similar (predict-observe-explain). Nonetheless, physics education research has found very diff… ▽ More Instructors in introductory physics courses often use labs and demonstrations to reinforce that the physics equations introduced in lectures and textbooks describe what actually happens in the real world. The surface features and instructional learning goals of such labs and demonstrations may seem quite similar (predict-observe-explain). Nonetheless, physics education research has found very different impacts of these instructional methods on students' physics content knowledge that depend critically on several instructional characteristics. In this paper, we disentangle the research by comparing details of the learning methods in verification labs, traditional lecture demonstrations, and enhanced lecture demonstrations. We discuss possible mechanisms for the measurably different learning outcomes by dissecting the activities according to the role of prediction, cognitive load, and engagement. We use this characterization to motivate rethinking the goals of labs towards experimentation skills and beliefs, rather than for verifying physics content. △ Less

Submitted 31 May, 2019; v1 submitted 8 December, 2017; originally announced December 2017.

Value added or misattributed? A multi-institution study on the educational benefit of labs for reinforcing physics content

Authors: N. G. Holmes, Jack Olsen, James L. Thomas, Carl E. Wieman

Abstract: Instructional labs are widely seen as a unique, albeit expensive, way to teach scientific content. We measured the effectiveness of introductory lab courses at achieving this educational goal across nine different lab courses at three very different institutions. These institutions and courses encompassed a broad range of student populations and instructional styles. The nine courses studied had t… ▽ More Instructional labs are widely seen as a unique, albeit expensive, way to teach scientific content. We measured the effectiveness of introductory lab courses at achieving this educational goal across nine different lab courses at three very different institutions. These institutions and courses encompassed a broad range of student populations and instructional styles. The nine courses studied had two key things in common: the labs aimed to reinforce the content presented in lectures, and the labs were optional. By comparing the performance of students who did and did not take the labs (with careful normalization for selection effects), we found universally and precisely no added value to learning from taking the labs as measured by course exam performance. This work should motivate institutions and departments to reexamine the goals and conduct of their lab courses, given their resource-intensive nature. We show why these results make sense when looking at the comparative mental processes of students involved in research and instructional labs, and offer alternative goals and instructional approaches that would make lab courses more educationally valuable. △ Less

Submitted 9 May, 2017; originally announced May 2017.

Comments: Accepted to Phys Rev PER

Journal ref: N. G. Holmes, Jack Olsen, James L. Thomas, and Carl E. Wieman, "Value added or misattributed? A multi-institution study on the educational benefit of labs for reinforcing physics content" Phys. Rev. Phys. Educ. Res. 13, 010129 (2017)

Toolboxes and handing students a hammer: The effects of cueing and instruction on getting students to think critically

Authors: N. G. Holmes, Dhaneesh Kumar, D. A. Bonn

Abstract: Developing critical thinking skills is a common goal of an undergraduate physics curriculum. How do students make sense of evidence and what do they do with it? In this study, we evaluated students' critical thinking behaviors through their written notebooks in an introductory physics laboratory course. We compared student behaviors in the Structured Quantitative Inquiry Labs (SQILabs) curriculum… ▽ More Developing critical thinking skills is a common goal of an undergraduate physics curriculum. How do students make sense of evidence and what do they do with it? In this study, we evaluated students' critical thinking behaviors through their written notebooks in an introductory physics laboratory course. We compared student behaviors in the Structured Quantitative Inquiry Labs (SQILabs) curriculum to a control group and evaluated the fragility of these behaviors through procedural cueing. We found that the SQILabs were generally effective at improving the quality of students' reasoning about data and making decisions from data. These improvements in reasoning and sensemaking were thwarted, however, by a procedural cue. We describe these changes in behavior through the lens of epistemological frames and task orientation, invoked by the instructional moves. △ Less

Submitted 20 March, 2017; originally announced March 2017.

Journal ref: N. G. Holmes, Dhaneesh Kumar, and D. A. Bonn, "Toolboxes and handing students a hammer: The effects of cueing and instruction on getting students to think critically", Phys. Rev. Phys. Educ. Res. 13, 010116 (2017)

Examining and contrasting the cognitive activities engaged in undergraduate research experiences and lab courses

Authors: N. G. Holmes, Carl E. Wieman

Abstract: While the positive outcomes of undergraduate research experiences (UREs) have been extensively categorized, the mechanisms for those outcomes are less understood. Through lightly structured focus group interviews, we have extracted the cognitive tasks that students identify as engaging in during their UREs. We also use their many comparative statements about their coursework, especially lab course… ▽ More While the positive outcomes of undergraduate research experiences (UREs) have been extensively categorized, the mechanisms for those outcomes are less understood. Through lightly structured focus group interviews, we have extracted the cognitive tasks that students identify as engaging in during their UREs. We also use their many comparative statements about their coursework, especially lab courses, to evaluate their experimental physics-related cognitive tasks in those environments. We find there are a number of cognitive tasks consistently encountered in physics UREs that are present in most experimental research. These are seldom encountered in lab or lecture courses, with some notable exceptions. Having time to reflect and fix or revise, and having a sense of autonomy, were both repeatedly cited as key enablers of the benefits of UREs. We also identify tasks encountered in actual experimental research that are not encountered in UREs. We use these findings to identify opportunities for better integration of the cognitive tasks in UREs and lab courses, as well as discussing the barriers that exist. This work responds to extensive calls for science education to better develop students' scientific skills and practices, as well as calls to expose more students to scientific research. △ Less

Submitted 5 July, 2016; originally announced July 2016.

Comments: 11 pages, 3 figures

Journal ref: Phys. Rev. Phys. Educ. Res. 12, 020103 (2016)

Teaching Critical Thinking

Authors: N. G. Holmes, Carl E. Wieman, D. A. Bonn

Abstract: The ability to make decisions based on data, with its inherent uncertainties and variability, is a complex and vital skill in the modern world. The need for such quantitative critical thinking occurs in many different contexts, and while it is an important goal of education, that goal is seldom being achieved. We argue that the key element for developing this ability is repeated practice in making… ▽ More The ability to make decisions based on data, with its inherent uncertainties and variability, is a complex and vital skill in the modern world. The need for such quantitative critical thinking occurs in many different contexts, and while it is an important goal of education, that goal is seldom being achieved. We argue that the key element for developing this ability is repeated practice in making decisions based on data, with feedback on those decisions. We demonstrate a structure for providing suitable practice that can be applied in any instructional setting that involves the acquisition of data and relating that data to scientific models. This study reports the results of applying that structure in an introductory physics lab course. Students in an experimental condition were repeatedly instructed to make and act on quantitative comparisons between datasets, and between data and models, an approach that is common to all science disciplines. These instructions were slowly faded across the course. After the instructions had been removed, students in the experimental condition were 12 times more likely to spontaneously propose or make changes to improve their experimental methods than a control group, who performed traditional experimental activities. They were also four times more likely to identify and explain a limitation of a physical model using their data. Students in the experimental condition also showed much more sophisticated reasoning about their data. These differences between the groups were seen to persist into a subsequent course taken the following year. △ Less

Submitted 19 August, 2015; originally announced August 2015.

Comments: Proceedings of the National Academy of Sciences (2015)

Measuring the Impact of Introductory Physics Labs on Learning

Authors: Carl Wieman, N. G. Holmes

Abstract: Our recent study showed that two lab courses, whose goals were exclusively to reinforce material developed in the lecture courses, do not have any impact on exam performance at the 1% level. In this study, we replicated this analysis with a modified version of one of these lab courses whose goals also included modeling, designing experiments, and analyzing and visualizing data. This modified cours… ▽ More Our recent study showed that two lab courses, whose goals were exclusively to reinforce material developed in the lecture courses, do not have any impact on exam performance at the 1% level. In this study, we replicated this analysis with a modified version of one of these lab courses whose goals also included modeling, designing experiments, and analyzing and visualizing data. This modified course used the same sets of apparatus as the previous version, but changed the pre-lab and in-lab activities to focus on developing and testing models with data. The study evaluated the impact of these additional goals and activities. We found that they did not affect students' performance on the final exam. △ Less

Submitted 1 July, 2015; originally announced July 2015.

Comments: Submitted to Physics Education Research Conference proceedings

Journal ref: American Journal of Physics, 83, 972 (2015)

Measuring the impact of an instructional laboratory on the learning of introductory physics

Authors: Carl Wieman, N. G. Holmes

Abstract: We have analyzed the impact of taking an associated lab course on the scores on final exam questions in two large introductory physics courses. Approximately a third of the students who completed each course also took an accompanying instructional lab course. The lab courses were fairly conventional, although they focused on supporting the mastery of a subset of the introductory physics topics cov… ▽ More We have analyzed the impact of taking an associated lab course on the scores on final exam questions in two large introductory physics courses. Approximately a third of the students who completed each course also took an accompanying instructional lab course. The lab courses were fairly conventional, although they focused on supporting the mastery of a subset of the introductory physics topics covered in the associated course. Performance between students who did and did not take the lab course was compared using final exam questions from the associated courses that related to concepts from the lab courses. The population of students who took the lab in each case was somewhat different from those who did not enroll in the lab course in terms of background and major. Those differences were taken into account by normalizing their performance on the lab-related questions with scores on the exam questions that did not involve material covered in the lab. When normalized in this way, the average score on lab-related questions of the students who took the lab, in both courses, was within 1% of the score of students who did not, with an uncertainty of 2%. This result raises questions as to the effectiveness of labs at supporting mastery of physics content. △ Less

Submitted 20 August, 2015; v1 submitted 9 June, 2015; originally announced June 2015.

Comments: Accepted for publication in the American Journal of Physics (August 2015)

Journal ref: Wieman and Holmes, "Measuring the impact of an instructional laboratory on the learning of introductory physics", American Journal of Physics 83, 972 (2015)

Quantitative comparisons to promote inquiry in the introductory physics lab

Authors: N. G. Holmes, D. A. Bonn

Abstract: In a recent report, the American Association of Physics Teachers has developed an updated set of recommendations for curriculum of undergraduate physics labs.1 This document focuses on six major themes: constructing knowledge, modeling, designing experiments, developing technical and practical laboratory skills, analyzing and visualizing data, and communicating physics. These themes all tie togeth… ▽ More In a recent report, the American Association of Physics Teachers has developed an updated set of recommendations for curriculum of undergraduate physics labs.1 This document focuses on six major themes: constructing knowledge, modeling, designing experiments, developing technical and practical laboratory skills, analyzing and visualizing data, and communicating physics. These themes all tie together as a set of practical skills in scientific measurement, analysis, and experimentation. In addition to teaching students how to use these skills, it is important for students to know when to use them so that they can use them autonomously. This requires, especially in the case of analytical skills, high-levels of inquiry behaviours to reflect on data and iterate measurements, which students rarely do in lab experiments. In this paper, we describe a simple framework for structuring the critical thinking and inquiry behaviours relevant to lab work, which focuses on iterative cycles of comparisons between data sets. We describe the framework and demonstrate examples of its use in an introductory physics lab course. △ Less

Submitted 9 June, 2015; originally announced June 2015.

Comments: Accepted for publication in The Physics Teacher (accepted: May 2015; to appear: Fall 2015)