Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/148598
Title: An empirical evaluation on the interpretable methods on Malware analysis
Authors: Ang, Alvis Jie Kai
Keywords: Engineering::Computer science and engineering
Issue Date: 2021
Publisher: Nanyang Technological University
Source: Ang, A. J. K. (2021). An empirical evaluation on the interpretable methods on Malware analysis. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/148598
Project: SCSE20-0196
Abstract: With the upsurge of cybersecurity attacks in recent years, there is a demand for more complex and accurate Malware classifiers to take the limelight. For these complex models to be trusted and be deployed in the wild, it is necessary for the results of these complex models to be explainable and thus trusted. However, complex black box models are difficult to be explained accurately with existing explanation techniques as different explanation techniques may perform better under different conditions. This report empirically evaluates the performance of the two most popular explanation techniques, LIME and SHAP, on a XGBoost classifier that was trained to classify Malware. The XGBoost model makes use of unigram and bigram as training features. To evaluate the performance of LIME and SHAP on the XGBoost model, we investigate the effects of the top ranked features from both explanation techniques by detecting the Malware class probability before and after eliminating the top ranked feature. While this metric may be a simple one, the consistency of the results show that it is nevertheless an effective one. Additionally, our results also show that SHAP consistently performs better than LIME on our model. Further investigation reveals that features ranked highly by LIME fluctuates greatly, from features that impact the class probabilities greatly to little or no effect from the XGBoost classifier used. Overall, using the metric proposed, we can perform evaluation of various explanation techniques on complex black box models.
URI: https://hdl.handle.net/10356/148598
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
Ang_Jie_Kai_Alvis_FYP_Final.pdf
  Restricted Access
470.15 kBAdobe PDFView/Open

Page view(s)

84
Updated on Jan 24, 2022

Download(s)

2
Updated on Jan 24, 2022

Google ScholarTM

Check

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.