Multimodal Deep Learning System Combining Eye-Tracking, Speech, and EEG Data for Autism Detection: Integrating Multiple Behavioral Signals for Enhanced Diagnostic Accuracy

Posted: Mar 05, 2021

• Mia Smith
• Mia Taylor
• Michael Johnson

Abstract

Autism Spectrum Disorder (ASD) represents a complex neurodevelopmental condition characterized by challenges in social communication, restricted interests, and repetitive behaviors. The current diagnostic landscape for ASD relies heavily on clinical observation, parent interviews, and standardized assessment tools such as the Autism Diagnostic Observation Schedule (ADOS) and Autism Diagnostic Interview-Revised (ADI-R). While these methods have proven valuable, they suffer from several limitations including subjectivity, inter-rater variability, and significant delays in diagnosis that can impact early intervention outcomes. The average age of ASD diagnosis in the United States remains around 4-5 years, despite evidence that reliable detection is possible as early as 18-24 months. This diagnostic gap represents a critical challenge in the field of developmental psychiatry and underscores the need for more objective, quantitative approaches to ASD assessment. Recent advances in computational methods and sensing technologies have opened new possibilities for automated ASD detection. However, most existing computational approaches have focused on single modalities, such as analyzing only eye-tracking data or exclusively examining speech patterns. These unimodal systems fail to capture the multifaceted nature of ASD, which manifests across multiple behavioral and neurophysiological domains. The integration of complementary data sources represents a promising direction for enhancing diagnostic accuracy and developing more comprehensive assessment tools. This research introduces a novel multimodal deep learning framework that simultaneously processes eye-tracking, speech, and EEG data to create a holistic profile of ASD-related characteristics. Our approach is grounded in the understanding that ASD affects multiple interconnected systems, including visual attention, language processing, and neural connectivity. By leveraging recent developments in multimodal fusion techniques and cross-modal learning, we have developed a system that not only achieves high diagnostic accuracy but also provides insights into the relative contributions of different behavioral