Multimodal Deep Learning System Combining Eye-Tracking, Speech, and EEG Data for Autism Detection: Integrating Multiple Behavioral Signals for Enhanced Diagnostic Accuracy
Posted: Aug 18, 2023
Abstract
The complex and heterogeneous nature of autism spectrum disorder necessitates diagnostic approaches that capture its multifaceted behavioral and neurophysiological manifestations. This research presents a novel multimodal deep learning system that integrates eye-tracking patterns, speech characteristics, and electroencephalography (EEG) data to achieve comprehensive autism detection. Our
framework employs specialized neural architectures for each modality: a temporal
convolutional network for eye-tracking gaze patterns, a transformer-based model
for speech prosody and linguistic features, and a graph neural network for EEG
functional connectivity. The system was developed and validated using a diverse
cohort of 1,250 participants aged 4-17 years, including 680 individuals with autism
spectrum disorder and 570 neurotypical controls. The integrated multimodal approach achieved exceptional performance with 96.3% accuracy, 95.8% sensitivity,
and 96.7% specificity, significantly outperforming unimodal approaches and existing
screening methods. Feature importance analysis revealed that eye-tracking social
attention patterns contributed most strongly to classification accuracy (42% relative
importance), followed by EEG gamma-band connectivity (31%) and speech prosody
features (27%). The system demonstrated robust generalizability across age groups
and sex, with consistent performance maintained in cross-validation with independent datasets. This research represents a significant technical advancement in
autism diagnostics by providing a quantitative, multimodal assessment framework
that captures the complex interplay between visual social processing, communication patterns, and neural synchrony characteristics of autism spectrum disorder.
Downloads: 321
Abstract Views: 2182
Rank: 274568