Auditing buffer overflow vulnerabilities using program analysis and data mining techniques

Abstract

This thesis presents approaches for auditing Buffer Overflow (BO) vulnerabilities, one of the highly prevalent and dangerous vulnerabilities from source code as well as x86 executables. While many approaches exist in literature that help in mitigating them, continuous presence of BO bugs in vulnerability reports suggests possible limitations in existing approaches or difficulty in their adoption. Therefore, alternative solutions which are effective and easy-to-use are needed to comprehensively address them. It is also imperative to devise mechanisms for auditing BO bugs from executables. Based on these observations, in this thesis, we propose three novel approaches for auditing BO vulnerabilities namely: test case generation, hybrid auditing methodology using static-dynamic analysis and machine learning for addressing vulnerabilities in source code and vulnerability prediction using static code attributes for predicting bugs in x86 executables. The thesis also evaluates the proposed approaches and demonstrates that they are useful and effective.