Efficient Visual Self-Attention
The attention mechanism is arguably one of the most important breakthroughs in deep learning in the last decade. It first appeared as an auxiliary module to assist for word alignment in machine translation. Later, the Transformer architecture revolutionarily replaced recurrence completely by self-attention and swiftly took over the entire field of natural language processing.
Its adoption in computer vision did not come until recently, with the quadratic computational complexity plaguing its applications. This talk dives deeply into a series of works by Mr. Shen on a novel efficient formulation of attention, its application onto video understanding, and the quest for a fully-attentional architecture on the basis of it.
Lecture slides: https://docs.google.com/presentation/...
00:00 Intro
03:39 Motivation for attention
08:01 Dot-product attention
13:41 Efficient attention
21:39 Empirical comparison vs. the non-local module
23:59 Semi-supervised video object segmentation
28:39 Space-time memory module
33:25 Deep learning approaches
37:26 Global context module
42:48 Empirical results
44:31 Global Self-Attention Networks
47:51 Motivations for fully-attentional modeling
01:00:58 Axial attention
01:02:03 GSA module
01:04:46 Discussion
Talk is based on the speaker's papers:
1. Efficient attention: https://arxiv.org/abs/1812.01243 ; https://github.com/cmsflash/efficient...
2. Global context module: https://arxiv.org/abs/2001.11243
3. GSA-Net: https://arxiv.org/abs/2010.03019
Presenter BIO:
Mr. Zhuoran Shen holds a BEng in Computer Science from The University of Hong Kong. He is joining Pony.ai as a Software Engineer in Perception. Earlier, he has been an AI Resident at Google Research and Research Interns at Tencent and SenseTime. His research focuses on the attention mechanism for computer vision, including fully-attentional visual modeling and efficient attention. He also has interests in large-scale visual pertaining and applications of computer vision.
His page: https://cmsflash.github.io/
-------------------------
Find us at:
Newsletter for updates about more events ➜ http://eepurl.com/gJ1t-D
Sub-reddit for discussions ➜ / 2d3dai
Discord server for, well, discord ➜ / discord
Blog ➜ https://2d3d.ai