Autoencoders – Ep. 10 (Deep Learning SIMPLIFIED)


There are times when it’s extremely useful
to figure out the underlying structure of a data set. Having access to the most important
data features gives you a lot of flexibility when you start applying labels. Autoencoders
are an important family of neural networks that are well-suited for this task. Let’s
take a look. In a previous video we looked at the Restricted
Boltzmann Machine, which is a very popular example of an autoencoder. But there are other
types of autoencoders like denoising and contractive, just to name a few. Just like an RBM, an autoencoder
is a neural net that takes a set of typically unlabelled inputs, and after encoding them,
tries to reconstruct them as accurately as possible. As a result of this, the net must
decide which of the data features are the most important, essentially acting as a feature
extraction engine. Autoencoders are typically very shallow, and
are usually comprised of an input layer, an output layer and a hidden layer. An RBM is
an example of an autoencoder with only two layers. Here is a forward pass that ends with
a reconstruction of the input. There are two steps – the encoding and the decoding. Typically,
the same weights that are used to encode a feature in the hidden layer are used to reconstruct
an image in the output layer. Autoencoders are trained with backpropagation,
using a metric called “loss”. As opposed to “cost”, loss measures the amount of
information that was lost when the net tried to reconstruct the input. A net with a small
loss value will produce reconstructions that look very similar to the originals. Not all of these nets are shallow. In fact,
deep autoencoders are extremely useful tools for dimensionality reduction. Consider an
image containing a 28×28 grid of pixels. A neural net would need to process over 750
input values just for one image – doing this across millions of images would waste
significant amounts of memory and processing time. A deep autoencoder could encode this
image into an impressive 30 numbers, and still maintain information about the key image features.
When decoding the output, the net acts like a two-way translator. In this example, a well-trained
net could translate these 30 encoded numbers back into a reconstruction that looks similar
to the original image. Certain types of nets also introduce random noise to the encoding-decoding
process, which has been shown to improve the robustness of the resulting patterns. Have you ever needed to use an autoencoder
to reduce the dimensionality of your data? If so, please comment and share your experiences. Deep autoencoders perform better at dimensionality
reduction than their predecessor, principal component analysis, or PCA. Below is a comparison
of two letter codes for news stories of different topics – generated by both a deep autoencoder
and a PCA. Labels were added to the picture for illustrative purposes. In the next video, we’ll take a look at
Recursive Neural Tensor Nets or RNTNs

16 Replies to “Autoencoders – Ep. 10 (Deep Learning SIMPLIFIED)”

  1. Great set of videos, I am really enjoying them. Hope that they get a bit hands on, where we can try out things that we have learnt on our own problems.

  2. No, I have never needed to use autoencoders to reduce dimensionality of data. But I would definitely use it to generate (reconstruct) dialogs. It is similar to what humans do with empathetic listening trying to understand somebody else. Perfect algorithm for self learning chatting bot!

  3. It looks like the animation in the video is out of sync with the voice over. It seems to stay on the RBM for a very long time, while mentioning other stuff that sounds like it should be in an animation. Otherwise though, very good stuff!

  4. Thanks for this wonderful lectures, I had a question:
    While using the Auto encoders, as mentioned in the video, we are reducing a 780 pixels input in to, say 30 numbers which accurately approximates the large input.. But the point which I don't understand is why it is required to again decode it to original Input/image? Since our purpose is either classification or regression..??

  5. I've used Discrete Cosine Transform to perform lossless compression of images and then re-construct them. Can I say that I used an Auto encoder?

  6. In EP.5, she says the vanishing gradient is a serious problem for deep learning neural network training task, and until 2006, Hinton, Lecun and Bengio have done some breakthrough work to conquer this vanishing gradient problem. But in the following videos, she introduced the CNN,RBM,DBN, autoencoder, does it mean we use these models to deal with the vanishing gradient problem? She didn't mention this conclusion in the video clearly. Is my understanding right?

  7. These video lectures are useful. The effort is commendable. I have a question about PCA. Can we reconstruct the data after reducing dimensions( By applying the PCA) ? If yes then how, please.

  8. I am looking forward to reduce the processing time of stacked autoencoder. Can you provide more information precise and specifics on stacked auto encoder please

Leave a Reply

Your email address will not be published. Required fields are marked *