Multimodal Sequence Classification of force-based instrumented hand manipulation motions using LSTM-RNN deep learning models

Abstract

The advent of mobile ubiquitous computing enabled sensor informatics of human movements to be used in modeling and building deep learning classifiers for cognitive AI. Expanding deep learning approaches for classifying instrumented hand manipulation tasks, especially the art of manual therapy and soft tissue manipulation, can potentially augment practitioner’s performance and enhance fidelity with computer assisted guidelines. This paper introduces a dataset of 3D force profiles and manipulation motion sequences of controlled soft tissue manipulation stroke pattern applications in thoracolumbar, upper thigh and calf regions of a single human subject performed by five experienced manual therapists. The multimodal 3D force, 3D accelerometer and resultant gyro raw data were preprocessed and experimentally fed into a multilayered Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) deep learning model to observe sequence classifications of two manipulation motion techniques (Linear "Strumming" motion and curvilinear "J-Stroke" arched motion) of manual therapy performed using a handheld, localizing Quantifiable Soft Tissue Manipulation (QSTM) medical tool. Each of these motion sequences were further labeled with corresponding best practice technique from validated video tapes and reclassified into "Correct" and "Incorrect" practice based on defined criteria. The deep learning model resulted in 90-95% classification accuracy for individual intra-therapist reduced dataset. The classification accuracy varied between 78%-93% range, when trained with multivariate characteristic feature set combinations for the complete spectrum of inter-therapist dataset.Clinical Relevance — AI informed online therapeutic guidelines can be leveraged to minimize practice inconsistencies, optimize educational training of therapy using data informed protocols, and study progression of pain and healing towards advancing manual therapy.