Browsing by Author "Sittig, Dean F."

Now showing 1 - 5 of 5
  • Thumbnail Image
    Item
    A Systematic Approach to Configuring MetaMap for Optimal Performance
    (Thieme, 2022) Jing, Xia; Indani, Akash; Hubig, Nina; Min, Hua; Gong, Yang; Cimino, James J.; Sittig, Dean F.; Rennert, Lior; Robinson, David; Biondich, Paul; Wright, Adam; Nøhr, Christian; Law, Timothy; Faxvaag, Arild; Gimbel, Ronald; Pediatrics, School of Medicine
    Background: MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward. Objective: To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance. Methods: MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated. Results: The percentages of exact matches and missing gold-standard terms were 0.6-0.79 and 0.09-0.3 for one behavior option, and 0.56-0.8 and 0.09-0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively. Conclusion: We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.
  • Thumbnail Image
    Item
    Dashboards for visual display of patient safety data: A systematic review
    (BMJ, 2021) Murphy, Daniel R.; Savoy, April; Satterly, Tyler; Sittig, Dean F.; Singh, Hardeep
    Background Methods to visualise patient safety data can support effective monitoring of safety events and discovery of trends. While quality dashboards are common, use and impact of dashboards to visualise patient safety event data remains poorly understood. Objectives To understand development, use and direct or indirect impacts of patient safety dashboards. Methods We conducted a systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched PubMed, EMBASE and CINAHL for publications between 1 January 1950 and 30 August 2018 involving use of dashboards to display data related to safety targets defined by the Agency for Healthcare Research and Quality’s Patient Safety Net. Two reviewers independently reviewed search results for inclusion in analysis and resolved disagreements by consensus. We collected data on development, use and impact via standardised data collection forms and analysed data using descriptive statistics. Results Literature search identified 4624 results which were narrowed to 33 publications after applying inclusion and exclusion criteria and consensus across reviewers. Publications included only time series and case study designs and were inpatient focused and emergency department focused. Information on direct impact of dashboards was limited, and only four studies included informatics or human factors principles in development or postimplementation evaluation. Discussion Use of patient-safety dashboards has grown over the past 15 years, but impact remains poorly understood. Dashboard design processes rarely use informatics or human factors principles to ensure that the available content and navigation assists task completion, communication or decision making. Conclusion Design and usability evaluation of patient safety dashboards should incorporate informatics and human factors principles. Future assessments should also rigorously explore their potential to support patient safety monitoring including direct or indirect impact on patient safety.
  • Thumbnail Image
    Item
    Keyphrase Identification Using Minimal Labeled Data with Hierarchical Context and Transfer Learning
    (medRxiv, 2023-05-26) Goli, Rohan; Hubig, Nina; Min, Hua; Gong, Yang; Sittig, Dean F.; Rennert, Lior; Robinson, David; Biondich, Paul; Wright, Adam; Nøhr, Christian; Law, Timothy; Faxvaag, Arild; Weaver, Aneesa; Gimbel, Ronald; Jing, Xia; Pediatrics, School of Medicine
    Interoperable clinical decision support system (CDSS) rules provide a pathway to interoperability, a well-recognized challenge in health information technology. Building an ontology facilitates creating interoperable CDSS rules, which can be achieved by identifying the keyphrases (KP) from the existing literature. However, KP identification for data labeling requires human expertise, consensus, and contextual understanding. This paper aims to present a semi-supervised KP identification framework using minimal labeled data based on hierarchical attention over the documents and domain adaptation. Our method outperforms the prior neural architectures by learning through synthetic labels for initial training, document-level contextual learning, language modeling, and fine-tuning with limited gold standard label data. To the best of our knowledge, this is the first functional framework for the CDSS sub-domain to identify KPs, which is trained on limited labeled data. It contributes to the general natural language processing (NLP) architectures in areas such as clinical NLP, where manual data labeling is challenging, and light-weighted deep learning models play a role in real-time KP identification as a complementary approach to human experts' effort.
  • Thumbnail Image
    Item
    Lessons Learned from Implementing Service-Oriented Clinical Decision Support at Four Sites: A Qualitative Study
    (Elsevier, 2015-11) Wright, Adam; Sittig, Dean F.; Ash, Joan S.; Erickson, Jessica L.; Hickman, Trang T.; Paterno, Marilyn; Gebhardt, Eric; McMullen, Carmit; Tsurikova, Ruslana; Dixon, Brian E.; Fraser, Greg; Simonaitis, Linas; Sonnenberg, Frank A.; Middleton, Blackford; Department of Epidemiology, Richard M. Fairbanks School of Public Health
    Objective To identify challenges, lessons learned and best practices for service-oriented clinical decision support, based on the results of the Clinical Decision Support Consortium, a multi-site study which developed, implemented and evaluated clinical decision support services in a diverse range of electronic health records. Methods Ethnographic investigation using the rapid assessment process, a procedure for agile qualitative data collection and analysis, including clinical observation, system demonstrations and analysis and 91 interviews. Results We identified challenges and lessons learned in eight dimensions: (1) hardware and software computing infrastructure, (2) clinical content, (3) human-computer interface, (4) people, (5) workflow and communication, (6) internal organizational policies, procedures, environment and culture, (7) external rules, regulations, and pressures and (8) system measurement and monitoring. Key challenges included performance issues (particularly related to data retrieval), differences in terminologies used across sites, workflow variability and the need for a legal framework. Discussion Based on the challenges and lessons learned, we identified eight best practices for developers and implementers of service-oriented clinical decision support: (1) optimize performance, or make asynchronous calls, (2) be liberal in what you accept (particularly for terminology), (3) foster clinical transparency, (4) develop a legal framework, (5) support a flexible front-end, (6) dedicate human resources, (7) support peer-to-peer communication, (8) improve standards. Conclusion The Clinical Decision Support Consortium successfully developed a clinical decision support service and implemented it in four different electronic health records and four diverse clinical sites; however, the process was arduous. The lessons identified by the Consortium may be useful for other developers and implementers of clinical decision support services.
  • Thumbnail Image
    Item
    The state of the art in clinical knowledge management: An inventory of tools and techniques
    (2010-01) Sittig, Dean F.; Wright, Adam; Simonaitis, Linas; Carpenter, James D.; Allen, George O.; Doebbeling, Bradley N.; Sirajuddin, Anwar Mohammad; Ash, Joan S.; Middleton, Blackford
    Purpose To explore the need for, and use of, high-quality, collaborative, clinical knowledge management (CKM) tools and techniques to manage clinical decision support (CDS) content. Methods In order to better understand the current state of the art in CKM, we developed a survey of potential CKM tools and techniques. We conducted an exploratory study by querying a convenience sample of respondents about their use of specific practices in CKM. Results The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and CDS implementations: (1) a multidisciplinary team responsible for creating and maintaining the clinical content; (2) an external organizational repository of clinical content with web-based viewer that allows anyone in the organization to review it; (3) an online, collaborative, interactive, Internet-based tool to facilitate content development; (4) an enterprise-wide tool to maintain the controlled clinical terminology concepts. Even organizations that have been successfully using computer-based provider order entry with advanced clinical decision support features for well over 15 years are not using all of the CKM tools or practices that we identified. Conclusions If we are to further stimulate progress in the area of clinical decision support, we must continue to develop and refine our understanding and use of advanced CKM capabilities.