1
$\begingroup$

I'm working with neural networks.

I have a number of inputs to the network which is far greater than the number of training instances ($\sim 7k$ inputs vs $700$ samples, $1$ order of magnitude, number of outputs $2k$).

My question, as a newcomer to neural nets, is this:

Should I diminish the size of the input layer (maybe through an autoencoder) ?

Dimensional reduction was suggested to me, but I didn't really understand why.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

A NN's inputs can be equal to number of features in data, and have less relationship with number of data points. For example, we can some DNA data, where we only have 1000 people / instances, but each instance has millions of features in DNA. In such setting, it is perfectly OK to build a NN with millions of inputs.

For your question about dimension reduction: the key idea of using NN is letting the model to figure out the feature engineering / necessary transformation. So, the model can automatically do the feature reduction to us. It is not very common to run feature reduction (say PCA) first then feed into NN, unless there are some computational resource constraints.

Also, output is 2K seems too much for me. Are you trying to predict a discrete outcome with 2K possible values? In most cases, people predicting a much less possible values, such as binary Yes/No. (There are some reasons on why it is hard to do to predict 2K possible values, which I will not explain here.)

Improve this answer
$\endgroup$
7
  • $\begingroup$ Sorry for bothering you, it's not that clear to me one thing: are you saying that the number of inputs ($7k$ in my case) is not necessarily equal to the number of features one extract from those inputs? Or are you plainly saying that a high number of inputs/features is not an issue whichever the number of data? I'm forecasting $2k$ products daily sales. For now I'm considering $340$ products (read output) and a corresponding $\sim 1k$ inputs. Forecasting performance minimizing WeightedMape out performs every other algorithm, so I was doubting the suggestion given to me $\endgroup$
    – Tommaso Guerrini
    Commented Feb 16, 2017 at 16:06
  • $\begingroup$ By the way (I wrote more here if you are interested: stats.stackexchange.com/questions/260444/…) $\endgroup$
    – Tommaso Guerrini
    Commented Feb 16, 2017 at 16:07
  • $\begingroup$ @TommasoGuerrini I am trying to say "high number of inputs/features is not an issue whichever the number of data" $\endgroup$
    – Haitao Du
    Commented Feb 16, 2017 at 16:07
  • 1
    $\begingroup$ @TommasoGuerrini yes, you could. And you can (and should) use regularization on your model. $\endgroup$
    – Haitao Du
    Commented Mar 8, 2017 at 15:01
  • 1
    $\begingroup$ @TommasoGuerrini adding regularization and reducing input dimensions are different things. that is why I said, the model will work well with huge number of inputs (with regularization.) $\endgroup$
    – Haitao Du
    Commented Mar 8, 2017 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.