Part-1 of our IJCAI tutorial on deep learning for Brain Encoding and Decoding
Subba Reddy Oota, Manish Gupta, Bapi Raju Surampudi, Mariya Toneva. Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding. The 32nd International Joint Conference on Artificial Intelligence (IJCAI-23). Aug 19th-Aug 25, 2023. Macao, S.A.R.
Slides: https://docs.google.com/presentation/...
How does the brain represent different modes of information? Can we design a system that automatically understands what the user is thinking? Such questions can be answered by studying brain recordings like functional magnetic resonance imaging (fMRI). As a first step, the neuroscience community has contributed several large cognitive neuroscience datasets related to passive reading/listening/viewing of concept words, narratives, pictures and movies. Encoding and decoding models using these datasets have also been proposed in the past two decades. These models serve as additional tools for basic research in cognitive science and neuroscience. Encoding models aim at generating fMRI brain representations given a stimulus automatically. They have several practical applications in evaluating and diagnosing neurological conditions and thus also help design therapies for brain damage. Decoding models solve the inverse problem of reconstructing the stimuli given the fMRI. They are useful for designing brain-machine or braincomputer interfaces. Inspired by the effectiveness of deep learning models for natural language processing, computer vision, and speech, recently several neural encoding and decoding models have been proposed. The latest ones leverage GPT-3, Wav2Vec2.0 and Stable Diffusion for processing text, speech and images respectively. In this tutorial, we plan to discuss different kinds of stimulus representations, and popular encoding and decoding architectures in detail. The tutorial will provide a working knowledge of the state of the art methods for encoding and decoding, a thorough understanding of the literature, and a better understanding of the benefits and limitations of encoding/decoding with deep learning.