Clustering with explanations

Abstract

Clustering is a powerful tool in data analysis as it enables the discovery of hidden patterns and group structures within complex datasets. In medical contexts, such as cancer screening, clustering techniques play a pivotal role in identifying distinct patient groups and supporting data driven decision making. Clustering can also be used to identify different subtypes within a particular disease. However, the opacity of traditional clustering algorithms often limits their utility in critical applications, where interpretability is paramount. This topic is also relatively underexplored as compared to the explainable methods used to explain supervised machine learning models like the Shapley Additive Explanations (SHAP) model. We explored various techniques and distance metrics to obtain useful clustering. These clusters are in line with the different subtypes a disease has. We then employ machine learning models and techniques suggested by other researchers to explain the clusters. This research focuses on the development of explainable clustering frameworks that combine advanced machine learning methodologies with interpretability to enhance transparency. By elucidating the underlying factors influencing cluster formation, we aim to provide actionable insights for medical professionals, improving patient stratification and early detection strategies. Our framework is validated using mainly irritable bowel syndrome (IBS) datasets and can be extended to other medical domains as well, demonstrating its potential for broad healthcare applications.