An Empirical Analysis of Vulnerability Detection Tools for Solidity Smart Contracts Using Line Level Manually Annotated Vulnerabilities
Authors:
Francesco Salzano,
Cosmo Kevin Antenucci,
Simone Scalabrino,
Giovanni Rosa,
Rocco Oliveto,
Remo Pareschi
Abstract:
The rapid adoption of blockchain technology highlighted the importance of ensuring the security of smart contracts due to their critical role in automated business logic execution on blockchain platforms. This paper provides an empirical evaluation of automated vulnerability analysis tools specifically designed for Solidity smart contracts. Leveraging the extensive SmartBugs 2.0 framework, which i…
▽ More
The rapid adoption of blockchain technology highlighted the importance of ensuring the security of smart contracts due to their critical role in automated business logic execution on blockchain platforms. This paper provides an empirical evaluation of automated vulnerability analysis tools specifically designed for Solidity smart contracts. Leveraging the extensive SmartBugs 2.0 framework, which includes 20 analysis tools, we conducted a comprehensive assessment using an annotated dataset of 2,182 instances we manually annotated with line-level vulnerability labels. Our evaluation highlights the detection effectiveness of these tools in detecting various types of vulnerabilities, as categorized by the DASP TOP 10 taxonomy. We evaluated the effectiveness of a Large Language Model-based detection method on two popular datasets. In this case, we obtained inconsistent results with the two datasets, showing unreliable detection when analyzing real-world smart contracts. Our study identifies significant variations in the accuracy and reliability of different tools and demonstrates the advantages of combining multiple detection methods to improve vulnerability identification. We identified a set of 3 tools that, combined, achieve up to 76.78\% found vulnerabilities taking less than one minute to run, on average. This study contributes to the field by releasing the largest dataset of manually analyzed smart contracts with line-level vulnerability annotations and the empirical evaluation of the greatest number of tools to date.
△ Less
Submitted 21 May, 2025;
originally announced May 2025.
Bridging the Gap: A Comparative Study of Academic and Developer Approaches to Smart Contract Vulnerabilities
Authors:
Francesco Salzano,
Lodovica Marchesi,
Cosmo Kevin Antenucci,
Simone Scalabrino,
Roberto Tonelli,
Rocco Oliveto,
Remo Pareschi
Abstract:
In this paper, we investigate the strategies adopted by Solidity developers to fix security vulnerabilities in smart contracts. Vulnerabilities are categorized using the DASP TOP 10 taxonomy, and fixing strategies are extracted from GitHub commits in open-source Solidity projects. Each commit was selected through a two-phase process: an initial filter using natural language processing techniques,…
▽ More
In this paper, we investigate the strategies adopted by Solidity developers to fix security vulnerabilities in smart contracts. Vulnerabilities are categorized using the DASP TOP 10 taxonomy, and fixing strategies are extracted from GitHub commits in open-source Solidity projects. Each commit was selected through a two-phase process: an initial filter using natural language processing techniques, followed by manual validation by the authors. We analyzed these commits to evaluate adherence to academic best practices. Our results show that developers often follow established guidelines for well-known vulnerability types such as Reentrancy and Arithmetic. However, in less-documented categories like Denial of Service, Bad Randomness, and Time Manipulation, adherence is significantly lower, suggesting gaps between academic literature and practical development. From non-aligned commits, we identified 27 novel fixing strategies not previously discussed in the literature. These emerging patterns offer actionable solutions for securing smart contracts in underexplored areas. To evaluate the quality of these new fixes, we conducted a questionnaire with academic and industry experts, who assessed each strategy based on Generalizability, Long-term Sustainability, and Effectiveness. Additionally, we performed a post-fix analysis by tracking subsequent commits to the fixed files, assessing the persistence and evolution of the fixes over time. Our findings offer an empirically grounded view of how vulnerabilities are addressed in practice, bridging theoretical knowledge and real-world solutions in the domain of smart contract security.
△ Less
Submitted 16 April, 2025;
originally announced April 2025.
