An evaluation of tokenizers on domain specific text

Abstract

The healthcare industry is fast realizing the value of data, collecting information from electronic health record systems (EHRs), sensors, and other sources. However, the problem of understanding the data collected in the process has been existed for years. According to big data analytics in healthcare, up to 80% of healthcare documentation is unstructured and hence generally unutilized, because mining and extracting this data is challenging and resource intensive. This is where Natural Language Processing can come in. NLP technology services have the potential to extract meaningful insights and concepts from data that was previously considered buried in text form. In NLP studies, text preprocessing is traditionally the first step in building a Machine Learning model, and in the process of text preprocessing, the very first and usually the most important step is tokenization. Currently, many open-source tools for tokenization are available for tokenizing text based on different rules, but few studies have been done on the performance of tokenizers on domain specific text—e.g., healthcare domain. Therefore, this project aims to, first, evaluate different open-source tokenizers’ performance on medical text data and select the best-performing tokenizer; after that, build a wrapper based on the best-performing tokenizer, to further improve its performance on medical text data. In this way, more accurate tokenization results of medical text data can be achieved, and these results can be used in the following NLP process to generate more meaningful insights. With NLP technology, physicians can enhance patient care, research efforts, and disease diagnosis methods.