1
$\begingroup$

Imagine we are working with the MNIST dataset and creating a neural network with 1 hidden layer. So we have a vector of 784 inputs, 100 hidden nodes, and 10 outputs.

If we were to visualize each node in the hidden layer (the wTx of inputs for each node) would we see features of the data? Such as circles, semi-circles, lines, etc?

And this is dimensionality reduction? Going from 784 pixels to 100 features

$\endgroup$
1
$\begingroup$

Erhan, Dumitru, et al. "Visualizing higher-layer features of a deep network." Dept. IRO, Université de Montréal, Tech. Rep 4323 (2009). APA demonstrates one way to visualize your hidden layers (and give examples on MNIST).

Going from 784 pixels to 100 features qualifies as dimensionality reduction.

$\endgroup$
1
$\begingroup$

Yes, it is dimensionality reduction, form your input dimension to number of your hidden networks. This is used in autoencoders, also you can throw away your output layer, and switch it for some other classifier, therefore using your NN as supervised feature learning unit.

Visualizing is however tricky, because feature representations are distributed across the network and they don't have spatial properties like SOM or more understandable dim. reduction like PCA.

They are however way's how to visualize what are they doing, or extract decision rules.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.