WeightWatcher, an Open-Source Diagnostic Tool for Analyzing Deep Neural Nets
Dr. Charles H. Martin, University of Chicago in Theoretical Chemical Physics, NSF Fellow, CalculationConsulting.com
TALK DESCRIPTION:
WeightWatcher (WW): is an open-source, diagnostic tool for analyzing Deep Neural Networks (DNN), without needing access to training or even test data. It can be used to:
analyze pre/trained PyTorch, Keras, DNN models (Conv2D and Dense layers)
monitor models, and the model layers, to see if they are over-trained or over-parameterized
predict test accuracies across different models, with or without training data
detect potential problems when compressing or fine-tuning pre-trained models
layer warning labels: over-trained; under-trained
as well as several new experimental model transformations, including:
SVDSmoothing: builds a model that can be used to predict test accuracies, but only with the training data.
SVDSharpness: removes Correlation Traps, which arise from sub-optimal regularization pre-trained models.
https://github.com/CalculatedContent/...
SPEAKER BIO:
I do machine learning, deep learning, data science, and AI software development, and with extensive domain experience in NLP for Search Relevance (as well as Text Generation and Quantitative Finance).
I have personally developed machine learning (ML) systems and helped get them into production at companies including Roche, France Telecom, GoDaddy, Aardvark (Google), eBay, eHow, Walmart, Barclays/BGI, and Blackrock. Recently I was both a consultant and FTE distinguished engineer at GLG, a very prestigious international consulting firm, where I developed AI methods for the search and recommendations platform.
I provide scientific consulting to the Page family office at the Anthropocene Institute, advising on areas of nuclear and quantum technologies with an eye toward climate change.
I do scientific research in collaboration with UC Berkeley on the foundations of AI and am the lead on the WeightWatcher project: pip install weightwatcher.
In 2011, I helped Demand Media / eHow become the first $1B IPO since Google: • C|C Demand Media Case Study
I was at Aardvark, acquired by Google and featured in the Lean StartUp, and a subject matter expert (SME) in ML at eBay.
At my first startup, in the late 90s, I developed semi-supervised ML algoRITHM for personalized search.
I offer over 15 years of commercial data science, software engineering, and ML experience. I am a full stack developer for web, object-oriented, and numerical programming. This includes java, ruby, python--and even dev ops.
I was coding on the Cray XMP in High School. I invented a Monte Carlo method for non-eq condensed matter systems and published--at 19. I am a national math contest winner.
My PhD is in Theoretical Chemistry from U Chicago. I was an NSF Fellow (1 of 2 nationwide).
/ charlesmartin14
For meeting details: https://www.meetup.com/SF-Bay-ACM/eve...
0:00 Chapter Intro
3:03 Talk & Speaker Intro
4:32 Presentation
5:46 Motivations: WeightWatcher Theory
7:15 Open source tool: weightwatcher
9:51 WeightWatcher: A diagnostic tool
11:20 Research: Implicit Self-Regularization in Deep Learning
12:49 WW: analyzes the ESD (eigenvalues) of the layer weight matrices
14:25 ESD of DNNs: detailed insight into W
15:55
19:14 GPT vs GPT-2
21:51 Heavy Tailed Metrics: GPT vs GPT2
22:57 Power Law Universality: ImageNet
24:09 Random Matrix Theory (RMT): detailed insight into W sub L
25:45 RMT: Marcenko Pastur
26:33 RMT: AlexNet
27:21 RMT: Heavy Tailed (HT)
28:06 HT: Self-Regularization (SR)
28:28 HT-SR Theory: 5+1 Phases of Training
30:43 Heavy Tailed RMT: Universality Classes
31:13 WW: predict trends in generalization
33:08 WW: Shape vs Scale metrics
35:04 WW: treat architecture changes
36:29 WW: predict test accuracies
37:02 Predicting test accuracies: Heavy tailed shape metrics
41:18 WW: detect overfitting?
57:57 WW: open-source, open-science
58:42 Q & A
1:11:45 Close