Umberfield, Elizabeth E.Jiang, YunFenton, Susan H.Stansbury, CooperFord, KathleenCrist, KayceeKardia, Sharon L. R.Thomer, Andrea K.Harris, Marcelline R.2024-03-262024-03-262021Umberfield EE, Jiang Y, Fenton SH, et al. Lessons Learned for Identifying and Annotating Permissions in Clinical Consent Forms. Appl Clin Inform. 2021;12(3):429-435. doi:10.1055/s-0041-1730032https://hdl.handle.net/1805/39530Background: The lack of machine-interpretable representations of consent permissions precludes development of tools that act upon permissions across information ecosystems, at scale. Objectives: To report the process, results, and lessons learned while annotating permissions in clinical consent forms. Methods: We conducted a retrospective analysis of clinical consent forms. We developed an annotation scheme following the MAMA (Model-Annotate-Model-Annotate) cycle and evaluated interannotator agreement (IAA) using observed agreement (A o), weighted kappa (κw ), and Krippendorff's α. Results: The final dataset included 6,399 sentences from 134 clinical consent forms. Complete agreement was achieved for 5,871 sentences, including 211 positively identified and 5,660 negatively identified as permission-sentences across all three annotators (A o = 0.944, Krippendorff's α = 0.599). These values reflect moderate to substantial IAA. Although permission-sentences contain a set of common words and structure, disagreements between annotators are largely explained by lexical variability and ambiguity in sentence meaning. Conclusion: Our findings point to the complexity of identifying permission-sentences within the clinical consent forms. We present our results in light of lessons learned, which may serve as a launching point for developing tools for automated permission extraction.en-USPublisher PolicyInformed consentConsent formsNatural language processingArticle