Deep Learning for Brain Encoding and Decoding
How does the brain represent different modes of information? Can we design a system that can automatically understand what the user is thinking? We can make progress towards answering such questions by studying brain recordings from devices such as functional magnetic resonance imaging (fMRI). The brain encoding problem aims to automatically generate fMRI brain representations given a stimulus. The brain decoding problem is the inverse problem of reconstructing the stimuli given the fMRI brain representation. Both the brain encoding and decoding problems have been studied in detail in the past two decades and the foremost attraction of studying these solutions is that they serve as additional tools for basic research in cognitive science and cognitive neuroscience. Recently, inspired by the effectiveness of deep learning models for natural language processing and computer vision, such models have been applied for neuroscience as well. In this tutorial, I discuss different kinds of stimulus representations, and popular encoding and decoding architectures in detail. The tutorial will provide a working knowledge of the state of the art methods for encoding and decoding, a thorough understanding of the literature, and a better understanding of the benefits and limitations of encoding/decoding with deep learning, along with related code.
Here is the agenda:
00:00:00 Introduction to Brain encoding and decoding
00:01:17 Brain Encoding/Decoding: Techniques and Research Goals
00:13:30 Introduction to popular datasets
00:16:20 Text Stimulus Datasets
00:21:35 Visual Stimulus Datasets
00:26:41 Audio Stimulus Datasets
00:29:09 Video Stimulus Datasets
00:31:32 Other Multi-modal Stimulus Datasets
00:34:47 Stimulus Representations
00:40:27 Text Stimulus Representations
00:59:20 Visual Stimulus Representations
01:04:13 Audio Stimulus Representations
01:07:02 Multimodal Stimulus Representations
01:12:50 Hands-on: https://tinyurl.com/DL4Brain
For more details, please look at https://github.com/jashna14/DL4Brain or https://docs.google.com/presentation/...
Tutorial: Subba Reddy Oota, Jashn Arora, Manish Gupta, Raju S. Bapi, Mariya Toneva. Deep Learning for Brain Encoding and Decoding. The 44th Annual Meeting of the Cognitive Science Society, Jul 27-30, 2022. Toronto, Canada.