-
Trusting code in the wild: Exploring contributor reputation measures to review dependencies in the Rust ecosystem
Authors:
Sivana Hamer,
Nasif Imtiaz,
Mahzabin Tamanna,
Preya Shabrina,
Laurie Williams
Abstract:
Developers rely on open-source packages and must review dependencies to safeguard against vulnerable or malicious upstream code. A careful review of all dependencies changes often does not occur in practice. Therefore, developers need signals to inform of dependency changes that require additional examination. The goal of this study is to help developers prioritize dependency review efforts by ana…
▽ More
Developers rely on open-source packages and must review dependencies to safeguard against vulnerable or malicious upstream code. A careful review of all dependencies changes often does not occur in practice. Therefore, developers need signals to inform of dependency changes that require additional examination. The goal of this study is to help developers prioritize dependency review efforts by analyzing contributor reputation measures as a signal. We use network centrality measures to proxy contributor reputation using collaboration activity. We employ a mixed method methodology from the top 1,644 packages in the Rust ecosystem to build a network of 6,949 developers, survey 285 developers, and model 5 centrality measures. We find that only 24% of respondents often review dependencies before adding or updating a package, mentioning difficulties in the review process. Additionally, 51% of respondents often consider contributor reputation when reviewing dependencies. The closeness centrality measure is a significant factor in explaining how developers review dependencies. Yet, centrality measures alone do not account for how developers choose to review dependencies. We recommend that ecosystems like GitHub, Rust, and npm implement a contributor reputation badge based on our modeled coefficients to aid developers in dependency reviews.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
Trusting code in the wild: A social network-based centrality rating for developers in the Rust ecosystem
Authors:
Nasif Imtiaz,
Preya Shabrina,
Laurie Williams
Abstract:
As modern software extensively uses open source packages, developers regularly pull in new upstream code through frequent updates. While a manual review of all upstream changes may not be practical, developers may rely on the authors' and reviewers' identities, among other factors, to decide what level of review the new code may require. The goal of this study is to help downstream project develop…
▽ More
As modern software extensively uses open source packages, developers regularly pull in new upstream code through frequent updates. While a manual review of all upstream changes may not be practical, developers may rely on the authors' and reviewers' identities, among other factors, to decide what level of review the new code may require. The goal of this study is to help downstream project developers prioritize review efforts for upstream code by providing a social network-based centrality rating for the authors and reviewers of that code. To that end, we build a social network of 6,949 developers across the collaboration activity from 1,644 Rust packages. Further, we survey the developers in the network to evaluate if code coming from a developer with a higher centrality rating is likely to be accepted with lesser scrutiny by the downstream projects and, therefore, is perceived to be more trusted. Our results show that 97.7\% of the developers from the studied packages are interconnected via collaboration, with each developer separated from another via only four other developers in the network. The interconnection among developers from different Rust packages establishes the ground for identifying the central developers in the ecosystem. Our survey responses ($N=206$) show that the respondents are more likely to not differentiate between developers in deciding how to review upstream changes (60.2\% of the time). However, when they do differentiate, our statistical analysis showed a significant correlation between developers' centrality ratings and the level of scrutiny their code might face from the downstream projects, as indicated by the respondents.
△ Less
Submitted 31 May, 2023;
originally announced June 2023.
-
The Power of Nudging: Exploring Three Interventions for Metacognitive Skills Instruction across Intelligent Tutoring Systems
Authors:
Mark Abdelshiheed,
John Wesley Hostetter,
Preya Shabrina,
Tiffany Barnes,
Min Chi
Abstract:
Deductive domains are typical of many cognitive skills in that no single problem-solving strategy is always optimal for solving all problems. It was shown that students who know how and when to use each strategy (StrTime) outperformed those who know neither and stick to the default strategy (Default). In this work, students were trained on a logic tutor that supports a default forward-chaining and…
▽ More
Deductive domains are typical of many cognitive skills in that no single problem-solving strategy is always optimal for solving all problems. It was shown that students who know how and when to use each strategy (StrTime) outperformed those who know neither and stick to the default strategy (Default). In this work, students were trained on a logic tutor that supports a default forward-chaining and a backward-chaining (BC) strategy, then a probability tutor that only supports BC. We investigated three types of interventions on teaching the Default students how and when to use which strategy on the logic tutor: Example, Nudge and Presented. Meanwhile, StrTime students received no interventions. Overall, our results show that Nudge outperformed their Default peers and caught up with StrTime on both tutors.
△ Less
Submitted 18 March, 2023;
originally announced March 2023.
-
A Multicriteria Evaluation for Data-Driven Programming Feedback Systems: Accuracy, Effectiveness, Fallibility, and Students' Response
Authors:
Preya Shabrina,
Samiha Marwan,
Andrew Bennison,
Min Chi,
Thomas Price,
Tiffany Barnes
Abstract:
Data-driven programming feedback systems can help novices to program in the absence of a human tutor. Prior evaluations showed that these systems improve learning in terms of test scores, or task completion efficiency. However, crucial aspects which can impact learning or reveal insights important for future improvement of such systems are ignored in these evaluations. These aspects include inhere…
▽ More
Data-driven programming feedback systems can help novices to program in the absence of a human tutor. Prior evaluations showed that these systems improve learning in terms of test scores, or task completion efficiency. However, crucial aspects which can impact learning or reveal insights important for future improvement of such systems are ignored in these evaluations. These aspects include inherent fallibility of current state-of-the-art, students' programming behavior in response to correct/incorrect feedback, and effective/ineffective system components. Consequently, a great deal of knowledge is yet to be discovered about such systems. In this paper, we apply a multi-criteria evaluation with 5 criteria on a data-driven feedback system integrated within a block-based novice programming environment. Each criterion in the evaluation reveals a unique pivotal aspect of the system: 1) How accurate the feedback system is; 2) How it guides students throughout programming tasks; 3) How it helps students in task completion; 4) What happens when it goes wrong; and 5) How students respond generally to the system. Our evaluation results showed that the system was helpful to students due to its effective design and feedback representation despite being fallible. However, novices can be negatively impacted by this fallibility due to high reliance and lack of self-evaluation. The negative impacts include increased working time, implementation, or submission of incorrect/partially correct solutions. The evaluation results reinforced the necessity of multi-criteria system evaluations while revealing important insights helpful to ensuring proper usage of data-driven feedback systems, designing fallibility mitigation steps, and driving research for future improvement.
△ Less
Submitted 26 July, 2022;
originally announced August 2022.
-
Investigating the Impact of Backward Strategy Learning in a Logic Tutor: Aiding Subgoal Learning towards Improved Problem Solving
Authors:
Preya Shabrina,
Behrooz Mostafavi,
Mark Abdelshiheed,
Min Chi,
Tiffany Barnes
Abstract:
Learning to derive subgoals reduces the gap between experts and students and makes students prepared for future problem solving. Researchers have explored subgoal labeled instructional materials with explanations in traditional problem solving and within tutoring systems to help novices learn to subgoal. However, only a little research is found on problem-solving strategies in relationship with su…
▽ More
Learning to derive subgoals reduces the gap between experts and students and makes students prepared for future problem solving. Researchers have explored subgoal labeled instructional materials with explanations in traditional problem solving and within tutoring systems to help novices learn to subgoal. However, only a little research is found on problem-solving strategies in relationship with subgoal learning. Also, these strategies are under-explored within computer-based tutors and learning environments. Backward problem-solving strategy is closely related to the process of subgoaling, where problem solving iteratively refines the goal into a new subgoal to reduce difficulty. In this paper, we explore a training strategy for backward strategy learning within an intelligent logic tutor that teaches logic proof construction. The training session involved backward worked examples (BWE) and problem-solving (BPS) to help students learn backward strategy towards improving their subgoaling and problem-solving skills. To evaluate the training strategy, we analyzed students' 1) experience with and engagement in learning backward strategy, 2) performance, and 3) proof construction approaches in new problems that they solved independently without tutor help after each level of training and in post-test. Our results showed that, when new problems were given to solve without any tutor help, students who were trained with both BWE and BPS outperformed students who received none of the treatment or only BWE during training. Additionally, students trained with both BWE and BPS derived subgoals during proof construction with significantly higher efficiency than the other two groups.
△ Less
Submitted 26 July, 2022;
originally announced August 2022.
