Authors: Meng Ren Zijing Yin Fuchen Ma Zhenyang Xu Yu Jiang Chengnian Sun Huizhong Li Yan Cai
Venue: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 566-579, 2021
Year: 2021
Abstract: Security of smart contracts has attracted increasing attention in recent years. Many researchers have devoted themselves to devising testing tools for vulnerability detection. Each published tool has demonstrated its effectiveness through a series of evaluations on their own experimental scenarios. However, the inconsistency of evaluation settings such as different data sets or performance metrics, may result in biased conclusion. In this paper, based on an empirical evaluation of widely used smart contract testing tools, we propose a unified standard to eliminate the bias in the assessment process. First, we collect 46,186 source-available smart contracts from four influential organizations. This comprehensive dataset is open to the public and involves different code characteristics, vulnerability patterns and application scenarios. Then we propose a 4-step evaluation process and summarize the difference among relevant work in these steps. We use nine representative tools to carry out extensive experiments. The results demonstrate that different choices of experimental settings could significantly affect tool performance and lead to misleading or even opposite conclusions. Finally, we generalize some problems of existing testing tools, and propose some possible directions for further improvement.
BibTeX:
@inproceedings{mengren2021eeosctwitbc,
author = "Meng Ren and Zijing Yin and Fuchen Ma and Zhenyang Xu and Yu Jiang and Chengnian Sun and Huizhong Li and Yan Cai",
title = "Empirical evaluation of smart contract testing: what is the best choice?",
year = "2021",
pages = "566-579",
booktitle = "Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis"
}
Plain Text:
Meng Ren, Zijing Yin, Fuchen Ma, Zhenyang Xu, Yu Jiang, Chengnian Sun, Huizhong Li, and Yan Cai, "Empirical evaluation of smart contract testing: what is the best choice?," 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 566-579