7
$\begingroup$

I have a training set of 87 samples and 9480 variables. My predictors are continuous and my response variable is binary. I'd like to use the caret package in R to tune a neural network classification model on my data. In order to do this, I first have to reduce the size of the predictor-set with feature selection to make it computationally feasible, correct?

Lets assume that is correct. Then the question becomes "which feature selection method?". If I were to do RF-RFE using the rfe function in caret, for example, I would get an optimal feature set specific to random forests and not necessarily optimal for neural networks.

Any advice?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ With only 87 observations, you are in a bind. How many positive & negative cases do you have? $\endgroup$
    – gung - Reinstate Monica
    Mar 4, 2015 at 16:59

3 Answers 3

Reset to default
19
$\begingroup$

One way to think about the process of building a predictive model (such as a neural network) is that you have a 'budget' of information to spend, much like a certain amount of money for a monthly household budget. With only 87 observations in your training set (and only 36 more in your test set), you have a very skimpy budget. In addition, there is much less information in a binary indicator (i.e., your predicted variable is positive vs. negative) than there is in a continuous variable. In truth, you may only have enough information to reliably estimate the proportion positive.

Neural networks have many advantages, but they require very large numbers of parameters to be estimated. When you have a hidden layer (or more than one hidden layer), and multiple input variables, the number of parameters (link weights) that need to be accurately estimated explodes. But every parameter to be estimated consumes some of your informational budget. You are essentially guaranteed to overfit this model (note that this has nothing to do with the computational feasibility of the algorithm). Unfortunately, I don't think cross-validation will get you out of these problems.

If you are committed to building a predictive model using your continuous variables, I would try a logistic regression model instead of a NN. It will use fewer parameters. I would fit the model with probably only one variable, or at most a couple, and use the test set to see if the additional variables (beyond the intercept only) create instability and reduce your out of sample accuracy.

Regarding the X variables themselves, I would use a method that is blind to the outcome. Specifically, I would try principal components analysis (PCA) and extract just the first one or two PCs. I honestly think this is going to be the best you are going to be able to do.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I do plan on trying out logistic regression as well...more models the better! PCA before NN is an intruiging idea, but have a few concerns. My worry is that maybe it's an old school way of doing it and that supervised multiariate feature selection would be better in order to lose less valuable information prior to modeling. Except if I do a supervised RF feature selection then NN, the features would be specific to RF, not NN. What about an "all relevant" FS such as Boruta, then do NN-RFE with an external repeated-cv (less variance than regular cv) resampling procedure to validate? $\endgroup$
    – Seldon
    Mar 5, 2015 at 17:48
  • 2
    $\begingroup$ I doubt "more models [is] better". My main point is that you don't have enough information to fit anything well. My first guess is that you only have enough information to fit an intercept (overall % positive), and should not use any of the variables. I think you are likely to end up w/ a model that is worse than nothing. It may help to read my answer here. $\endgroup$
    – gung - Reinstate Monica
    Mar 5, 2015 at 17:57
  • $\begingroup$ More models is better because part of my goal is to gain familiarity and experience with building different models, as well as finding the optimum. Also more models is better because part of my end goal is to examine the models' resampling distributions to find uncorrelated models that would make for good ensembles. True my data set suffers from the CoD badly, but so do almost all gene/protein expression datasets. Which is why PCA concerned me...can't afford to lose any of what little predictive ability my dataset has. $\endgroup$
    – Seldon
    Mar 5, 2015 at 18:50
  • $\begingroup$ Also your answer you linked was very well written and thought out, but disagree that it applies here. I'm not using stepwise or AIC, for one. Also, I'm tuning models based on a validation set using external resampling (see jstatsoft.org/v28/i05/paper), then making final decisions with a single final look at the test set. This should handle the randomness as best as possible. $\endgroup$
    – Seldon
    Mar 5, 2015 at 18:54
  • $\begingroup$ It's no problem. If you want experience making models, "more models is better" makes sense. You're also right that my linked answer was about stepwise selection w/o c-v. But as I stated in my answer above, I don't thin c-v will save you here (I could be wrong, though). It may help to read this. $\endgroup$
    – gung - Reinstate Monica
    Mar 5, 2015 at 19:18
0
$\begingroup$

With respect to your first question, it depends on your computer.

With respect to the second, there is no single best answer. Neural networks are themselves often used for feature selection. This is the paradigm leading to deep learning. In that case it is unlikely you'd want to do any feature selection (except maybe whitening of the data).

If you're fitting a shallow network via backprop and are worried about overfitting, doing the (unprincipled but often effective) PCA and dropping less important components might help you out.

Improve this answer
$\endgroup$
0
0
$\begingroup$

Adding my own two cents to the previous answers. You have a big problem with the curse of dimensionality. This problem with the tiny corpus you have makes me think that the best option in this case would be a simple Bayesian.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.