Researchers in natural language processing (NLP) have made significant progress recently. We have also witnessed more NLP applications in widespread use, such as chatbots, grammar checkers and more. One emerging application is medical text mining, which involves the use of NLP algorithms to process clinical notes written by doctors, to derive useful information and knowledge. Natural language provides a flexible and versatile form to record information. It is the most flexible form for doctors to document the symptoms, diagnoses and treatments of patients.
Professor Ng Hwee Tou (NUS Department of Computer Science) and his team, in collaboration with Dr. Ngiam Kee Yuan at the National University Hospital, came up with an automated NLP software for appendicitis diagnosis.
In acute appendicitis, the appendix is inflamed and needs to be removed by abdominal surgery. Acute appendicitis is a medical emergency and hence accurate and prompt diagnosis is crucial. Their NLP algorithm processes the clinical notes written by doctors in the emergency department. Based on the natural language description of a patient’s symptoms and conditions in a clinical note, their algorithm determines whether a patient suffers from acute appendicitis. The task is challenging since the clinical notes do not consist of well-formed complete sentences, but rather are written in sentence fragments with frequent misspelled words, abbreviations, acronyms, and medical jargon.
In 2017, Professor Ng’s team implemented a novel deep neural network for this diagnosis task, which is formulated as a binary classification task. They do not require manual annotation to collect their training data, since the ground truth diagnosis of a patient can be obtained from the patient’s discharge summary. In their approach, manual engineering of features is not required as the neural network learns the necessary representations that best predict the diagnosis decision entirely based on the training data. In addition, the attention layer in the neural network is able to highlight the important words in a clinical note that leads to the diagnosis decision. This serves as a simple form of explanation that justifies why the system comes to its diagnosis decision.
The empirical evaluation based on historical patient data indicates that the accuracy of their automated diagnosis tool is promising. They envisage that their automated diagnosis tool can serve as a decision aid to doctors. A further randomized controlled trial is underway to evaluate the effectiveness of doctors’ appendicitis diagnosis when aided by the automated diagnosis tool.
For more information on Prof Ng and his research work, please click here.