// Databricks notebook source exported at Tue, 28 Jun 2016 09:28:40 UTC
Scalable Data Science
prepared by Raazesh Sainudiin and Sivanand Sivaram
supported by 
and
The html source url of this databricks notebook and its recorded Uji 
:
Deep learning: A Crash Introduction
This notebook provides an introduction to Deep Learning. It is meant to help you descend more fully into these learning resources and references:
- Udacity's course on Deep Learning https://www.udacity.com/course/deep-learning--ud730 by Arpan Chakraborty and Vincent Vanhoucke
- Neural networks and deep learning http://neuralnetworksanddeeplearning.com/ by Michael Nielsen
- Deep learning book http://www.deeplearningbook.org/ by Ian Goodfellow, Yoshua Bengio and Aaron Courville
- Deep learning - buzzword for Artifical Neural Networks
- What is it?
- Supervised learning model - Classifier
- Unsupervised model - Anomaly detection
- Needs lots of data
- Online learning model - backpropogation
- Optimization - Stochastic gradient descent
- Regularization - L1, L2, Dropout
- Supervised
- Fully connected network
- Convolutional neural network - Eg: For classifying images
- Recurrent neural networks - Eg: For use on text, speech
- Unsupervised
- Autoencoder
A quick recap of logistic regression / linear models
(watch now 46 seconds):
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Regression
y = mx + c
Another way to look at a linear model
-- Image Credit: Michael Nielsen
Recap - Gradient descent
(1:54 seconds):
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Recap - Stochastic Gradient descent
(2:25 seconds):
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
HOGWILD! Parallel SGD without locks http://i.stanford.edu/hazy/papers/hogwild-nips.pdf
Why deep learning? - Linear model
(24 seconds):
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
ReLU - Rectified linear unit or Rectifier - max(0, x)
-- Image Credit: Wikipedia
Neural Network
Watch now (45 seconds)
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Is decision tree a linear model?
http://datascience.stackexchange.com/questions/6787/is-decision-tree-algorithm-a-linear-or-nonlinear-algorithm
Neural Network
-- Image credit: Wikipedia
Multiple hidden layers
-- Image credit: Michael Nielsen
What does it mean to go deep? What do each of the hidden layers learn?
Watch now (1:13 seconds)
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Chain rule
(f o g)' = (f' o g) . g'
Chain rule in neural networks
Watch later (55 seconds)
-- Video Credit: Udacity's deep learning by Arpan Chakraborthy and Vincent Vanhoucke
Backpropogation
Watch later (9:55 seconds)
Watch now (1: 54 seconds)
How do you set the learning rate? - Step size in SGD?
Convolutional Neural Networks
Watch (3:55)
- Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
- Convolutional Neural networks blog - http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
Recurrent neural network
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Watch (3:55)
LSTM - Long short term memory
GRU - Gated recurrent unit
http://arxiv.org/pdf/1406.1078v3.pdf
Autoencoder
Watch (3:51)
Scalable Data Science
prepared by Raazesh Sainudiin and Sivanand Sivaram
supported by
and