Accelerate your Behavioral Medicine Research Using Machine Learning

April Idalski Carcone, PhD; Behavioral Informatics and Technology (BIT) SIG

Chances are you’ve probably heard the term “machine learning”, but what exactly is it? And, why should behavioral scientists care about it? A recent article in Medium, an online publishing platform, described how computer scientists are leveraging behavioral science to increase website traffic and app usage, explaining behavioral science as “an interdisciplinary approach to understanding human behavior via anthropology, sociology and psychology”.¹ It’s time for behavioral scientists to integrate computer science, specifically machine learning, into our interdisciplinary mosaic. Here’s why.

Simply put, machine learning is a class of statistical analysis techniques in which a computer "learns" to recognize patterns in data without being explicitly programmed to do so. A familiar example of machine learning at work are those advertisements that populate your favorite social media site or web browser searches. Have you ever thought “Wait a minute… how does X know I shop at Y?!” Well, you were just browsing Y’s website allowing the algorithms programmed into X’s website to learn that you like to shop at Y. Using this knowledge, X’s algorithms will now target Y’s advertising to you. While in the context of internet browsing, the use of machine learning models may be helpful, annoying, or even disturbing, these models have powerful implications for behavioral science. For example, at the 2019 SBM Annual Meeting, behavioral scientists described how they used machine learning to interpret data from wearable sensors, such as quantifying screen time based on color detection or identifying eating patterns among participants with obesity. Others used machine learning models to predict eating disorder treatment outcomes from medical chart data and examine the relationship between affect and activity levels.

There are two broad classes of machine learning models – supervised and unsupervised. Supervised machine learning models use coded data (“training” data, in computer science speak) to learn a mathematical model to map inputs (raw data) to the result of a particular cognitive task, which is represented as a code or “label”. Through this process, computers mimic the decision-making process human coders use to perform the same task, with accuracy comparable to human coders² but at a fraction of the time required. However, this means that a supervised machine learning model is only as good as the data you use to train it. To illustrate, my colleague, Alexander Kotov PhD and I have developed a series of supervised machine learning models to assign behavioral codes to clinical transcripts of patient-provider communication. Human coders had previously coded transcripts of adolescent patients and their caregivers engaged in a weight loss intervention using a code scheme that operationalizes patient-provider clinical communication according to the Motivational Interviewing framework. We used this coded data to train a machine learning model to recognize and assign these behavioral codes.³ The inter-rater reliability between the man and the machine was k=.663, compared to k=.696 between human coders.⁴ We then tested the model in a novel clinical context, HIV clinical care.⁵ With no modification, our model accurately identified 70% of patient-provider behaviors (k = .663) compared to human coders. We are currently working to develop additional machine learning models to fully automate the behavioral coding process, including models to parse transcripts into codable segments⁶ and analyze the sequencing of communication behavior.⁷ This is just one example of how behavioral scientists have leveraged supervised machine learning models. Others have used this approach to analyze clinical transcripts in the assessment of treatment fidelity,⁸ biomedical data to enhance fMRI diagnostic prediction,9 and to predict mental health outcomes (e.g., suicide) from medical chart data¹⁰ and developmental outcomes (e.g., developmental language disorders) from screening instrument data.¹¹

Unsupervised machine learning models contrast with supervised models in that they use mathematical models to discover patterns in uncoded data.¹² The ability to use uncoded data is a real strength of unsupervised models because of the resource investment required to construct a training coded data set. However, with no training data to compare model performance to, a big drawback of unsupervised machine learning is the inability to assess the accuracy of the model. Rather, computer scientists rely on content area experts, such as behavioral scientists, to guide decisions regarding model accuracy. Some examples of unsupervised machine learning applications include identifying people at high risk for developing dementia from population-based survey data,¹³ patients at high risk for death due to chronic heart failure,¹⁴ and clinical subtypes of Chronic Obstructive Pulmonary Disease from electronic medical records.¹⁵

The primary strengths of machine learning models are their ability to process large amounts of data very efficiently and their ability to detect very complex patterns in data. Machine learning analytics far outstrip the abilities of even the best-trained team of human analysts giving them the potential to accelerate behavioral research to unprecedented rates. On the other hand, because models are developed from the data, an inherent weakness of machine learning models is the integrity of the data set used – change the data, change the model. Thus, the best models are built on large, high quality data sets, which can be difficult and resource intensive to construct. For example, the initial training data file in the Motivational Interviewing example above was composed of only 38 interactions and required nearly a year to code, after the year needed to develop the coding scheme and train the coders! Model interpretability can also be an issue. The logic underlying some models’ decision-making may not be easily traceable or follow a linear path, particularly in more complex unsupervised models which many have thousands, if not more, rules. This complexity makes it difficult to assess the accuracy of some machine learning models.

The examples mentioned here are just the tip of the iceberg. There are many applications of machine learning models with vast implications for both research and clinical care. The Human Behaviour Change Project is using machine learning to cull the published literature to inform behavior change intervention and theory. The Behavioral Research in Technology and Engineering (BRiTE) Center is home to several technology driven projects to improve mental health services. NIH’s Big Data to Knowledge (BD2K) program encourages researchers to engage in data-driven discovery using technology-based tools and strategies, like machine learning, to analyze the ever-growing corpus of complex biological data. These initiatives are the foundation of precision medicine.

So, what are you waiting for? Go find a computer scientist and start harnessing the power of the machine! Still unsure about how machine learning models might augment your work? In addition to the articles referenced here, SAS has a free downloadable white paper that is a great primer on machine learning and its many applications. It’s a quick read making it a great way to get started thinking about how machine learning might inform your work.

 

References

1. Hassan: Behavioral Science & AI: The Good, The Bad & The Ugly. Retrieved January 3, 2019, 2019 from https://medium.com/datadriveninvestor/behavioral-science-ai-the-good-the-bad-the-ugly-656d2312ce71
2. Bishop CM: Pattern Recognition and Machine Learning. New York: Springer, 2007.
3. Hasan M, Kotov A, Carcone AI, et al.: A study of the effectiveness of machine learning methods for classification of clinical interview fragments into a large number of categories. Journal of Biomedical Informatics. 2016, 62:21-31.
4. Idalski Carcone A, Naar-King S, Brogan K, et al.: Provider communication behaviors that predict motivation to change in African American adolescents with obesity. Journal of Development and Behavioral Pediatrics. 2013, 34:599-608.
5. Idalski Carcone A, Hasan M, Alexander G, et al.: Developing Machine-Learning Models for Behavioral Coding. Journal of Pediatric Psychology. in press.
6. Hasan M, Kotov A, Naar S, Alexander GL, Idalski Carcone A: Deep Neural Architectures for Discourse Segmentation in E-Mail Based Behavioral Interventions. American Medical Informatics Association (AMIA) 2019 Informatics Summit. San Francisco, CA: American Medical Informatics Association, 2019.
7. Hasan M, Carcone AI, Naar S, et al.: Identifying Effective Motivational Interviewing Communication Sequences Using Automated Pattern Analysis. Journal of Health Informatics. in press:1-21.
8. Atkins DC, Steyvers M, Imel ZE, Smyth P: Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification. Implementation Science. 2014, 9:1-11.
9. Deshpande G, Li Z, Santhanam P, et al.: Recursive cluster elimination based support vector machine for disease state prediction using resting state functional and effective brain connectivity. PloS one. 2010, 5:e14277.
10. Adamou M, Antoniou G, Greasidou E, et al.: Mining Free-Text Medical Notes for Suicide Risk Assessment. Proceedings of the 10th Hellenic Conference on Artificial Intelligence. Patras, Greece: ACM, 2018, 1-8.
11. Armstrong R, Symons M, Scott J, et al.: Predicting Language Difficulties in Middle Childhood From Early Developmental Milestones: A Comparison of Traditional Regression and Machine Learning Techniques. Journal of speech, language, and hearing research: JSLHR. 2018:1.
12. Hofmann T: Unsupervised Learning by Probabilistic Latent Semantic Analysis. Machine Learning. 2001, 42:177-196.
13. Cleret de Langavant L, Bayen E, Yaffe K: Unsupervised Machine Learning to Identify High Likelihood of Dementia in Population-Based Surveys: Development and Validation Study. Journal of medical Internet research. 2018, 20:e10493-e10493.
14. Tokodi M, Schwertner W, Perge P, et al.: Unsupervised Machine Learning Algorithm to Identify High and Low Risk Patients Following CRT Implantation. Journal of the American College of Cardiology. 2018, 71:A947.
15. Pikoula M, Quint J, Direk K, et al.: Discovery of Novel COPD Subtypes Via Cluster Analysis of Clinical Features Extracted from Electronic Health Records. D13. IMPROVING DIAGNOSIS, CARE QUALITY, AND ADHERENCE IN COPD: American Thoracic Society, 2018, A6170-A6170.