notes, thoughts, and practice of applied machine learning

223 subscribers

0 articles/week

We introduce the weightwatcher (ww) , a python tool for computing quality metrics of trained, and pretrained, Deep Neural Networks pip install weightwatcher Here is an example with pretrained VGG11 from pytorch (ww works with keras models also): import weightwatcher as ww import torchvision.models as models model = models.vgg11(pretrained=True)...

Feb 2020

Introduction For the past year or two, we have talked a lot about how we can understand the properties of Deep Neural Networks by examining the spectral properties of the layer weight matrices . Specifically, we can form the correlation matrix , and compute the eigenvalues . By plotting the histogram of the eigenvalues...

Dec 2019

Big thanks to and the team at This Week in Machine Learning and AI for my recent interview: https://twimlai.com/meetups/implicit-self-regularization-in-deep-neural-networks/

Apr 2019

My Collaborator did a great job giving a talk on our research at the local San Francisco Bay ACM Meetup Michael W. Mahoney UC Berkeley Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models and smaller models trained from...

Apr 2019

My talk at ICSI-the International Computer Science Institute at UC Berkeley. ICSI is a leading independent, nonprofit center for research in computer science. Why Deep Learning Works: Self Regularization in Neural Networks Presented Thursday, December 13, 2018 The slides are available on my slideshare. The supporting tool, WeightWatcher, can be...

Dec 2018

This is a followup to a previous post: DON’T PEEK: DEEP LEARNING WITHOUT LOOKING … AT TEST DATA The idea…suppose we want to compare 2 or more deep neural networks (DNNs). Maybe we are fine tuning a DNN for transfer learning, or comparing a new architecture to an old on, or we are just tuning our hyper-parameters. Can we determine which DNN will...

Nov 2018

Machine Learning and AI for the Lean Start Up My recent talk at the French Tech Hub Startup Accelerator

Nov 2018

What is the purpose of a theory ? To explain why something works. Sure. But what good is a theory (i.e VC) that is totally useless in practice ? A good theory makes predictions. Recently we introduced the theory of Implicit Self-Regularization in Deep Neural Networks. Most notably, we observe that in all pre-trained models, the layer weight matrices...

Oct 2018

We can learn a lot about Why Deep Learning Works by studying the properties of the layer weight matrices of pre-trained neural networks. And, hopefully, by doing this, we can get some insight into what a well trained DNN looks like–even without peaking at the training data. One broad question we can ask is: How is information concentrated in Deep...

Sep 2018

Power Law Distributions in Deep Learning In a previous post, we saw that the Fully Connected (FC) layers of the most common pre-trained Deep Learning display power law behavior. Specifically, for each FC weight matrix , we compute the eigenvalues of the correlation matrix For every FC matrix, the eigenvalue frequencies, or Empirical Spectral...

Sep 2018

Subscribe to RSS Feeds, Blogs, Podcasts, Twitter searches, Facebook pages, even Email Newsletters! Get unfiltered news feeds or filter them to your liking.

Get Inoreader