Goulet | Seminar | How do neural networks learn?
Presenter: James-A Goulet | Professor, Polytechnique Montreal
Title: How do neural networks learn? A microscopic look into hidden units through Bayesian inference.
Abstract: We are surrounded by applications of neural networks which are capable of achieving complex tasks with an astonishing simplicity. In their basic form, neural networks are known to be universal functions approximators, i.e., they can approximate arbitrary complex functions. In the case of simple feedforward architectures using ReLU activation functions, this approximation is piecewise linear with the number of segments being function of the number of hidden units. With the recent developments related to the TAGI method, we have shown how we can use approximate Gaussian inference as the inference engine for arbitrary large neural networks. Despite the substantial reduction in the number of epochs required to train networks, it remains that even with TAGI, the parameters of neural networks cannot be learnt online in a single epoch. In this presentation, I will present the preliminary work done to identify the root cause of this limitation. The strategy is to proceed from the simplest neural network configuration possible and then build up in order to identify at what point online learning stops to be possible and why it is so.
Seminar archive: http://profs.polymtl.ca/jagoulet/Site...