Browsing by Subject "Natural language processing"

Now showing 1 - 10 of 30
  • Thumbnail Image
    Item
    A Systematic Approach to Configuring MetaMap for Optimal Performance
    (Thieme, 2022) Jing, Xia; Indani, Akash; Hubig, Nina; Min, Hua; Gong, Yang; Cimino, James J.; Sittig, Dean F.; Rennert, Lior; Robinson, David; Biondich, Paul; Wright, Adam; Nøhr, Christian; Law, Timothy; Faxvaag, Arild; Gimbel, Ronald; Pediatrics, School of Medicine
    Background: MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward. Objective: To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance. Methods: MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated. Results: The percentages of exact matches and missing gold-standard terms were 0.6-0.79 and 0.09-0.3 for one behavior option, and 0.56-0.8 and 0.09-0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively. Conclusion: We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.
  • Thumbnail Image
    Item
    Accuracy of a Commercial Large Language Model (ChatGPT) to Perform Disaster Triage of Simulated Patients Using the Simple Triage and Rapid Treatment (START) Protocol: Gage Repeatability and Reproducibility Study
    (JMIR, 2024-09-30) Franc, Jeffrey Micheal; Hertelendy, Attila Julius; Cheng, Lenard; Hata, Ryan; Verde, Manuela; Emergency Medicine, School of Medicine
    Background: The release of ChatGPT (OpenAI) in November 2022 drastically reduced the barrier to using artificial intelligence by allowing a simple web-based text interface to a large language model (LLM). One use case where ChatGPT could be useful is in triaging patients at the site of a disaster using the Simple Triage and Rapid Treatment (START) protocol. However, LLMs experience several common errors including hallucinations (also called confabulations) and prompt dependency. Objective: This study addresses the research problem: "Can ChatGPT adequately triage simulated disaster patients using the START protocol?" by measuring three outcomes: repeatability, reproducibility, and accuracy. Methods: Nine prompts were developed by 5 disaster medicine physicians. A Python script queried ChatGPT Version 4 for each prompt combined with 391 validated simulated patient vignettes. Ten repetitions of each combination were performed for a total of 35,190 simulated triages. A reference standard START triage code for each simulated case was assigned by 2 disaster medicine specialists (JMF and MV), with a third specialist (LC) added if the first two did not agree. Results were evaluated using a gage repeatability and reproducibility study (gage R and R). Repeatability was defined as variation due to repeated use of the same prompt. Reproducibility was defined as variation due to the use of different prompts on the same patient vignette. Accuracy was defined as agreement with the reference standard. Results: Although 35,102 (99.7%) queries returned a valid START score, there was considerable variability. Repeatability (use of the same prompt repeatedly) was 14% of the overall variation. Reproducibility (use of different prompts) was 4.1% of the overall variation. The accuracy of ChatGPT for START was 63.9% with a 32.9% overtriage rate and a 3.1% undertriage rate. Accuracy varied by prompt with a maximum of 71.8% and a minimum of 46.7%. Conclusions: This study indicates that ChatGPT version 4 is insufficient to triage simulated disaster patients via the START protocol. It demonstrated suboptimal repeatability and reproducibility. The overall accuracy of triage was only 63.9%. Health care professionals are advised to exercise caution while using commercial LLMs for vital medical determinations, given that these tools may commonly produce inaccurate data, colloquially referred to as hallucinations or confabulations. Artificial intelligence-guided tools should undergo rigorous statistical evaluation-using methods such as gage R and R-before implementation into clinical settings.
  • Thumbnail Image
    Item
    Advanced natural language processing and temporal mining for clinical discovery
    (2015-08-17) Mehrabi, Saeed; Jones, Josette F.; Palakal, Mathew J.; Chien, Stanley Yung-Ping; Liu, Xiaowen; Schmidt, C. Max
    There has been vast and growing amount of healthcare data especially with the rapid adoption of electronic health records (EHRs) as a result of the HITECH act of 2009. It is estimated that around 80% of the clinical information resides in the unstructured narrative of an EHR. Recently, natural language processing (NLP) techniques have offered opportunities to extract information from unstructured clinical texts needed for various clinical applications. A popular method for enabling secondary uses of EHRs is information or concept extraction, a subtask of NLP that seeks to locate and classify elements within text based on the context. Extraction of clinical concepts without considering the context has many complications, including inaccurate diagnosis of patients and contamination of study cohorts. Identifying the negation status and whether a clinical concept belongs to patients or his family members are two of the challenges faced in context detection. A negation algorithm called Dependency Parser Negation (DEEPEN) has been developed in this research study by taking into account the dependency relationship between negation words and concepts within a sentence using the Stanford Dependency Parser. The study results demonstrate that DEEPEN, can reduce the number of incorrect negation assignment for patients with positive findings, and therefore improve the identification of patients with the target clinical findings in EHRs. Additionally, an NLP system consisting of section segmentation and relation discovery was developed to identify patients' family history. To assess the generalizability of the negation and family history algorithm, data from a different clinical institution was used in both algorithm evaluations.
  • Thumbnail Image
    Item
    Annotating and Detecting Topics in Social Media Forum and Modelling the Annotation to Derive Directions-A Case Study
    (Research Square, 2021) B., Athira; Jones, Josette; Idicula, Sumam Mary; Kulanthaivel, Anand; Zhang, Enming; BioHealth Informatics, School of Informatics and Computing
    The widespread influence of social media impacts every aspect of life, including the healthcare sector. Although medics and health professionals are the final decision makers, the advice and recommendations obtained from fellow patients are significant. In this context, the present paper explores the topics of discussion posted by breast cancer patients and survivors on online forums. The study examines an online forum, Breastcancer.org, maps the discussion entries to several topics, and proposes a machine learning model based on a classification algorithm to characterize the topics. To explore the topics of breast cancer patients and survivors, approximately 1000 posts are selected and manually labeled with annotations. In contrast, millions of posts are available to build the labels. A semi-supervised learning technique is used to build the labels for the unlabeled data; hence, the large data are classified using a deep learning algorithm. The deep learning algorithm BiLSTM with BERT word embedding technique provided a better f1-score of 79.5%. This method is able to classify the following topics: medication reviews, clinician knowledge, various treatment options, seeking and providing support, diagnostic procedures, financial issues and implications for everyday life. What matters the most for the patients is coping with everyday living as well as seeking and providing emotional and informational support. The approach and findings show the potential of studying social media to provide insight into patients' experiences with cancer like critical health problems.
  • Thumbnail Image
    Item
    Automated pancreatic cyst screening using natural language processing: a new tool in the early detection of pancreatic cancer
    (Elsevier, 2015-05) Roch, Alexandra M.; Mehrabi, Saeed; Krishnan, Anand; Schmidt, Heidi E.; Kesterson, Joseph; Beesley, Chris; Dexter, Paul R.; Palakal, Matthew; Schmidt, C. Max; Department of Surgery, IU School of Medicine
    INTRODUCTION: As many as 3% of computed tomography (CT) scans detect pancreatic cysts. Because pancreatic cysts are incidental, ubiquitous and poorly understood, follow-up is often not performed. Pancreatic cysts may have a significant malignant potential and their identification represents a 'window of opportunity' for the early detection of pancreatic cancer. The purpose of this study was to implement an automated Natural Language Processing (NLP)-based pancreatic cyst identification system. METHOD: A multidisciplinary team was assembled. NLP-based identification algorithms were developed based on key words commonly used by physicians to describe pancreatic cysts and programmed for automated search of electronic medical records. A pilot study was conducted prospectively in a single institution. RESULTS: From March to September 2013, 566,233 reports belonging to 50,669 patients were analysed. The mean number of patients reported with a pancreatic cyst was 88/month (range 78-98). The mean sensitivity and specificity were 99.9% and 98.8%, respectively. CONCLUSION: NLP is an effective tool to automatically identify patients with pancreatic cysts based on electronic medical records (EMR). This highly accurate system can help capture patients 'at-risk' of pancreatic cancer in a registry.
  • Thumbnail Image
    Item
    Biomedical concept association and clustering using word embeddings
    (2018-12) Shah, Setu; Luo, Xiao; El-Sharkawy, Mohamed; King, Brian
    Biomedical data exists in the form of journal articles, research studies, electronic health records, care guidelines, etc. While text mining and natural language processing tools have been widely employed across various domains, these are just taking off in the healthcare space. A primary hurdle that makes it difficult to build artificial intelligence models that use biomedical data, is the limited amount of labelled data available. Since most models rely on supervised or semi-supervised methods, generating large amounts of pre-processed labelled data that can be used for training purposes becomes extremely costly. Even for datasets that are labelled, the lack of normalization of biomedical concepts further affects the quality of results produced and limits the application to a restricted dataset. This affects reproducibility of the results and techniques across datasets, making it difficult to deploy research solutions to improve healthcare services. The research presented in this thesis focuses on reducing the need to create labels for biomedical text mining by using unsupervised recurrent neural networks. The proposed method utilizes word embeddings to generate vector representations of biomedical concepts based on semantics and context. Experiments with unsupervised clustering of these biomedical concepts show that concepts that are similar to each other are clustered together. While this clustering captures different synonyms of the same concept, it also captures the similarities between various diseases and the symptoms that those diseases are symptomatic of. To test the performance of the concept vectors on corpora of documents, a document vector generation method that utilizes these concept vectors is also proposed. The document vectors thus generated are used as an input to clustering algorithms, and the results show that across multiple corpora, the proposed methods of concept and document vector generation outperform the baselines and provide more meaningful clustering. The applications of this document clustering are huge, especially in the search and retrieval space, providing clinicians, researchers and patients more holistic and comprehensive results than relying on the exclusive term that they search for. At the end, a framework for extracting clinical information that can be mapped to electronic health records from preventive care guidelines is presented. The extracted information can be integrated with the clinical decision support system of an electronic health record. A visualization tool to better understand and observe patient trajectories is also explored. Both these methods have potential to improve the preventive care services provided to patients.
  • Thumbnail Image
    Item
    DEEPEN: A negation detection system for clinical text incorporating dependency relation into NegEx
    (Elsevier, 2015-04) Mehrabi, Saeed; Krishnan, Krishnan; Sohn, Sunghwan; Roch, Alexandra M; Schmidt, Heidi; Kesterson, Joe; Beesley, Chris; Dexter, Paul; Schmidt, C. Max; Liu, Hongfang; Palakal, Mathew; Surgery, School of Medicine
    In Electronic Health Records (EHRs), much of valuable information regarding patients’ conditions is embedded in free text format. Natural language processing (NLP) techniques have been developed to extract clinical information from free text. One challenge faced in clinical NLP is that the meaning of clinical entities is heavily affected by modifiers such as negation. A negation detection algorithm, NegEx, applies a simplistic approach that has been shown to be powerful in clinical NLP. However, due to the failure to consider the contextual relationship between words within a sentence, NegEx fails to correctly capture the negation status of concepts in complex sentences. Incorrect negation assignment could cause inaccurate diagnosis of patients’ condition or contaminated study cohorts. We developed a negation algorithm called DEEPEN to decrease NegEx’s false positives by taking into account the dependency relationship between negation words and concepts within a sentence using Stanford dependency parser. The system was developed and tested using EHR data from Indiana University (IU) and it was further evaluated on Mayo Clinic dataset to assess its generalizability. The evaluation results demonstrate DEEPEN, which incorporates dependency parsing into NegEx, can reduce the number of incorrect negation assignment for patients with positive findings, and therefore improve the identification of patients with the target clinical findings in EHRs.
  • Thumbnail Image
    Item
    Designing a Predictive Coding System for Electronic Discovery
    (2016-04-08) Soundarajan, Dhivya; Hook, Sara Anne
    Not long ago, the concept of using predictive coding and other technologies to assist with the electronic discovery process seemed revolutionary. Da Silva Moore and Global Aerospace stand as the first major cases where judges strongly supported predictive coding.1-2 A recent Indiana case recognized it as a useful method for reducing the amount of potentially relevant evidence that has to be searched and culled.3 Within just a few short years, using predictive coding as part of an electronic discovery process is now considered acceptable and perhaps even expected. It is not difficult to appreciate the advantages of predictive coding and its superiority over a manual process at various steps of electronic discovery, particularly during the review step.4-11 However, questions still remain about the efficacy of the predictive coding process and the tools that are available.12-13 Because the use of predictive coding systems in law is still in its infancy, it presents us with an opportunity to design something that will not only take advantage of the power of big data and computational algorithms, but that will also incorporate design and usability principles to provide an attractive and easy-to-use interface for lawyers to interact with. Predictive coding uses natural language processing and other mathematical models to enhance search results, but the essence of these systems is that they actually learn and the precision of the retrieval improves as additional collections of evidence are entered. Behind-the-scenes will be a repository where all of the evidence for a case resides. Our system will assist the lawyers in reducing the time and cost of an electronic discovery process as well as minimize the chances for mistakes in determining which evidence is relevant to a case and which evidence can be withheld under attorney-client privilege, as attorney work-product or another confidentiality doctrine. 1. Da Silva Moore v. Publicis Groupe & MSL Group, No. 11 Civ. 1279, 2012 WL 607412 (ALC) (AJP) (S.D.N.Y. Feb. 24, 2012). 2. Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL 61040 (Vir. Cir. Ct. Apr. 23, 2012). 3. In re Biomet, 2013 WL 1729682 (N.D. Ind. Apr. 18, 2013). 4. Alison Silverstein and Geoffrey Vance. E-Discovery Myth Busters: Why Predictive Coding is Safe, Successful and Smart. Peer to Peer, Vol. 29, No. 4, December 2013, pp. 66-69. 5. John Papageorge. Predictive Coding Gaining Support in Courts. Indiana Lawyer, January 29-February 11, 2014, p. 8. 6. Adam M. Acosta. Predictive Coding: The Beginning of a New E-Discovery Era. Res Gestae, October 2012, pp. 8-14. 7. Ajith (AJ) Samuel. Analytics Driving the E-Discovery Process. Peer to Peer, Vol. 28, No. 2, June 2012. 8. Richard Acello. Beyond Prediction: Technology-Assisted Review Enters the Lexicon. ABA Journal, August 2012, pp. 37, 70. 9. Barry Murphy. The Rise of Technology-Assisted Review (TAR). Peer to Peer, Vol. 28, No. 2, June 2012, pp. 10. Brian Ingram. Controlling E-Discovery Costs in a Big Data World. Peer to Peer, Vol. 29, No. 1, March 2013. 11. Hal Marcus and Susan Stone. Beyond Predictive Coding - The True Power of Data Analytics [webinar]. International Legal Technology Association, May 19, 2015. 12. Jessica Watts and Gareth Evans. Predictive Coding in the Real World [webinar]. International Legal Technology Association, August 5, 2015. 13. Danielle Bethea. Predictive Coding: Revolutionizing Review or Still Gaining Momentum? Litigation and Practice Support: ITLA White Paper, International Legal Technology Association, June 2014.
  • Thumbnail Image
    Item
    Developing a dynamic recommendation system for personalizing educational content within an E-learning network
    (2018) Mirzaeibonehkhater, Marzieh; King, Brian; Jafari, Ali; Liu, Hongbo
    This research proposed a dynamic recommendation system for a social learning environment entitled CourseNetworking (CN). The CN provides an opportunity for the users to satisfy their academic requirement in which they receive the most relevant and updated content. In our research, we extracted some implicit and explicit features from the system, which are the most relevant user feature and posts features. The selected features are used to make a rating scale between users and posts so that represent the link between user and post in this learning management system (LMS). We developed an algorithm which measures the link between each user and post for the individual. To achieve our goal in our system design, we applied natural language processing technique (NLP) for text analysis and applied various classi cation technique with the aim of feature selection. We believe that considering the content of the posts in learning environments as an impactful feature will greatly affect to the performance of our system. Our experimental results demonstrated that our recommender system predicts the most informative and relevant posts to the users. Our system design addressed the sparsity and cold-start problems, which are the two main challenging issues in recommender systems.
  • Thumbnail Image
    Item
    Development and validation of a pragmatic natural language processing approach to identifying falls in older adults in the emergency department
    (Biomed Central, 2019-07-22) Patterson, Brian W.; Jacobsohn, Gwen C.; Shah, Manish N.; Song, Yiqiang; Maru, Apoorva; Venkatesh, Arjun K.; Zhong, Monica; Taylor, Katherine; Hamedani, Azita G.; Mendonça, Eneida A.; Pediatrics, IU School of Medicine
    BACKGROUND: Falls among older adults are both a common reason for presentation to the emergency department, and a major source of morbidity and mortality. It is critical to identify fall patients quickly and reliably during, and immediately after, emergency department encounters in order to deliver appropriate care and referrals. Unfortunately, falls are difficult to identify without manual chart review, a time intensive process infeasible for many applications including surveillance and quality reporting. Here we describe a pragmatic NLP approach to automating fall identification. METHODS: In this single center retrospective review, 500 emergency department provider notes from older adult patients (age 65 and older) were randomly selected for analysis. A simple, rules-based NLP algorithm for fall identification was developed and evaluated on a development set of 1084 notes, then compared with identification by consensus of trained abstractors blinded to NLP results. RESULTS: The NLP pipeline demonstrated a recall (sensitivity) of 95.8%, specificity of 97.4%, precision of 92.0%, and F1 score of 0.939 for identifying fall events within emergency physician visit notes, as compared to gold standard manual abstraction by human coders. CONCLUSIONS: Our pragmatic NLP algorithm was able to identify falls in ED notes with excellent precision and recall, comparable to that of more labor-intensive manual abstraction. This finding offers promise not just for improving research methods, but as a potential for identifying patients for targeted interventions, quality measure development and epidemiologic surveillance.