=======================
Deep Learning Tutorials
=======================

Deep Learning is a new area of Machine Learning research, which
has been introduced with the objective of moving Machine Learning
closer to one of its original goals: Artificial Intelligence.
See these course notes for a `brief introduction to Machine Learning for AI <http://www.iro.umontreal.ca/~pift6266/H10/notes/mlintro.html>`_
and an `introduction to Deep Learning algorithms <http://www.iro.umontreal.ca/~pift6266/H10/notes/deepintro.html>`_.

Deep Learning is about learning multiple levels of representation
and abstraction that help to
make sense of data such as images, sound, and text. 
For more about deep learning algorithms, see for example:

 - The monograph or review paper `Learning Deep Architectures for AI <http://www.iro.umontreal.ca/~lisa/publications2/index.php/publications/show/239>`_ (Foundations & Trends in Machine Learning, 2009).
 - The ICML 2009 Workshop on Learning Feature Hierarchies `webpage <http://www.cs.toronto.edu/~rsalakhu/deeplearning/index.html>`_ has a `list of references <http://www.cs.toronto.edu/~rsalakhu/deeplearning/references.html>`_.
 - The LISA `public wiki <http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/WebHome>`_ has a `reading list <http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/ReadingOnDeepNetworks>`_ and a `bibliography <http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/DeepNetworksBibliography>`_.
 - Geoff Hinton has `readings <http://www.cs.toronto.edu/~hinton/deeprefs.html>`_ from 2009's `NIPS tutorial <http://videolectures.net/jul09_hinton_deeplearn/>`_.

The tutorials presented here will introduce you to some of the most important deep learning
algorithms and will also show you how to run them using Theano_. Theano is a python library that makes writing deep learning models easy, and gives the option of
training them on a GPU.

The algorithm tutorials have some prerequisites.  You should know some python,
and be familiar with numpy. Since this tutorial is about using Theano, you
should read over the `Theano basic tutorial`_ first.  Once you've done that,
read through our :ref:`gettingstarted` chapter -- it introduces the notation, and downloadable datasets used in the algorithm tutorials, and the way we do optimization by stochastic gradient descent.

The code is available on the `Deep Learning Tutorial repositories <https://github.com/lisa-lab/DeepLearningTutorials>`_.

The purely supervised learning algorithms are meant to be read in order:

  #. :ref:`Logistic Regression <logreg>` - using Theano for something simple
  #. :ref:`Multilayer perceptron <mlp>` - introduction to layers
  #. :ref:`Deep Convolutional Network <lenet>` - a simplified version of LeNet5

The unsupervised and semi-supervised learning algorithms can be read in any
order (the auto-encoders can be read independently of the RBM/DBN thread):

  * :ref:`Auto Encoders, Denoising Autoencoders <daa>` - description of autoencoders
  * :ref:`Stacked Denoising Auto-Encoders <SdA>` - easy steps into unsupervised pre-training for deep nets
  * :ref:`Restricted Boltzmann Machines <rbm>` - single layer generative RBM model
  * :ref:`Deep Belief Networks <DBN>` - unsupervised generative pre-training of stacked RBMs followed by supervised fine-tuning

Building towards including the mcRBM model, we have a new tutorial on sampling
from energy models:

  * :ref:`HMC Sampling <HMC>` - hybrid (aka Hamiltonian) Monte-Carlo sampling with scan()

Building towards including the Contractive auto-encoders tutorial, we have the code for now:
  * `Contractive auto-encoders`_ code - There is some basic doc in the code.

Recurrent neural networks with word embeddings and context window:
  * :ref:`Semantic Parsing of Speech using Recurrent Net <rnnslu>`

LSTM network for sentiment analysis:
  * :ref:`LSTM network <lstm>`

Energy-based recurrent neural network (RNN-RBM):
  * :ref:`Modeling and generating sequences of polyphonic music <rnnrbm>`

.. _Theano: http://deeplearning.net/software/theano

.. _Theano basic tutorial: http://deeplearning.net/software/theano/tutorial

.. _Contractive auto-encoders: https://github.com/lisa-lab/DeepLearningTutorials/blob/master/code/cA.py

Note that the tutorials here are all compatible with Python 2 and 3, with the exception of :ref:`rnnrbm` which is only available for Python 2.