Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models

Authors: Baljinder Ghotra Shane McIntosh Ahmed E. Hassan

Venue: ICSE   2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, pp. 789-800, 2015

Year: 2015

Abstract: Defect prediction models help software quality assurance teams to effectively allocate their limited resources to the most defect-prone software modules. A variety of classification techniques have been used to build defect prediction models ranging from simple (e.g., Logistic regression) to advanced techniques (e.g., Multivariate Adaptive Regression Splines (MARS)). Surprisingly, recent research on the NASA dataset suggests that the performance of a defect prediction model is not significantly impacted by the classification technique that is used to train it. However, the dataset that is used in the prior study is both: (a) noisy, i.e., Contains erroneous entries and (b) biased, i.e., Only contains software developed in one setting. Hence, we set out to replicate this prior study in two experimental settings. First, we apply the replicated procedure to the same (known-to-be noisy) NASA dataset, where we derive similar results to the prior study, i.e., The impact that classification techniques have appear to be minimal. Next, we apply the replicated procedure to two new datasets: (a) the cleaned version of the NASA dataset and (b) the PROMISE dataset, which contains open source software developed in a variety of settings (e.g., Apache, GNU). The results in these new datasets show a clear, statistically distinct separation of groups of techniques, i.e., The choice of classification technique has an impact on the performance of defect prediction models. Indeed, contrary to earlier research, our results suggest that some classification techniques tend to produce defect prediction models that outperform others.

BibTeX:

@inproceedings{baljinderghotra2015rtioctotpodpm,
    author = "Baljinder Ghotra and Shane McIntosh and Ahmed E. Hassan",
    title = "Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models",
    year = "2015",
    pages = "789-800",
    booktitle = "Proceedings of 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering"
}

Plain Text:

Baljinder Ghotra, Shane McIntosh, and Ahmed E. Hassan, "Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models," 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, pp. 789-800