0
$\begingroup$

Model: Assume we want to learn patterns using an autoencoder neural network. In the simplest case, such a network is "shallow" with 1 hidden layer, takes a $d$-dimensional numerical input vector $x$, feeds it through the network and compares $x$ with the resulting $\tilde{x}$ at the output layer, where some information is lost between $x$ and $\tilde{x}$ due to e.g. a 'bottleneck' in the hidden layer (smaller number of neurons than in the input and output layer, forcing data compression). It is trained to reconstruct $x$ at the output layer by minimizing the mean-squared-error (MSE) between $x$ and $\tilde{x}$. After training, samples with small error would be "in line" with learned patterns, while larger errors would point to deviations.

Typical Application: This technique is often applied to images with a relatively large input vector dimension $d$. For example, the MNIST dataset of handwritten digits will have input vector size $d=28\times 28=784$ for every image.

Question: However, this technique could serve many other pattern recognition tasks, most of which have likely lower dimensional input size. Is there a minimum size of $d$ for the technique to work? For example, would an input layer with size $d\leq10$ still be feasible, where the data would be numerical time series data?

$\endgroup$
0
$\begingroup$

There is no minimum dimension for an autoencoder, you could have even 2 dimensions being embedded in one, and this would work.

Do note that the simplest autoencoder has 2 layers, not 3, one to connect the input to the embedded representation, and one to connect the embedded representation to the output.

$\endgroup$
  • $\begingroup$ Thanks, I changed my initial post regarding the layers. Does the same hold for time series data which is quite noisy in nature? For example, if we consider time series of hourly client purchase activity in US-dollar for 10 clients ($x$ with $d=10$) and we have $n=5000$ of such vectors forming our training set, which are quite noisy in nature, would the methodology still work in the same way? $\endgroup$ – robot_2077198 Dec 27 '18 at 21:21
  • $\begingroup$ You mean only 10 samples, each of them with 5000 dimensions? $\endgroup$ – Matthieu Brucher Dec 27 '18 at 21:23
  • $\begingroup$ The other way around, 5000 samples each being a 10-dimensional vector that are then fed one after the other through the network. $\endgroup$ – robot_2077198 Dec 27 '18 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.