Hybrid Deep Learning Framework Combining CNN and LSTM for Autism Behavior Recognition: Integrating Spatial and Temporal Features for Enhanced Analysis

Posted: Sep 19, 2019

• Hammad Khan
  Punjab College
• Jacob Williams
  University of Illinois Urbana-Champaign
• Olivia Brown
  University of California Los Angeles

Abstract

Autism Spectrum Disorder (ASD) is characterized by complex behavioral patterns that manifest across both spatial and temporal dimensions, presenting significant challenges for automated recognition systems. This research introduces a
novel hybrid deep learning framework that synergistically combines Convolutional
Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to capture both spatial features from visual data and temporal dynamics from behavioral
sequences. Our architecture processes video data of social interactions through
parallel CNN streams for spatial feature extraction from individual frames, coupled with LSTM networks that model temporal dependencies across behavioral
sequences. The framework incorporates multi-scale attention mechanisms, adaptive fusion techniques, and hierarchical feature aggregation to effectively integrate
spatial and temporal information. We evaluated our approach on a comprehensive
dataset of 2,300 video sequences from 850 children aged 24-60 months, including
both structured assessment sessions and naturalistic interactions. The proposed hybrid model achieved 93.7% recognition accuracy, significantly outperforming standalone CNN (86.2%) and LSTM (82.4%) approaches. Feature importance analysis
revealed that the integration of gaze pattern spatial features with temporal dynamics of social responsiveness provided the most discriminative power for autism
behavior recognition. This research demonstrates that the synergistic combination
of spatial and temporal modeling enables more accurate and clinically meaningful
autism behavior analysis, providing a robust foundation for computer-aided diagnostic systems and intervention monitoring tools.