I have just 25 observations. I'm not sure would it possible to test & train the data. For example 15 observations for train and 10 observations for test set. 15 observations is so small for training! I considered permutation feature selection and LOOCV with elastic-net. It left 11 features out of 1000. I used LOOCV in random forest modelling. So I modeled 15 observations with11 features. I got good results but I'm not sure I'm overfitting or not
12
-
5$\begingroup$ The answer I linked to advised against using feature selection methods, and concluded that you should collect more data or select variables based on theory (so not an algorithm or an automated method). I think the advice still applies. Perhaps other people will have additional recommendations. I'd suggest to edit your question to add all the info you mentioned in your previous comment, as many users of this website overlook comments. $\endgroup$– J-J-JCommented Aug 12 at 10:01
-
1$\begingroup$ could you please explain more about "select variables based on theory"? Thanks in advance for your time and help $\endgroup$– Leila aliCommented Aug 12 at 10:09
-
2$\begingroup$ It means based on subject matter knowledge, so it requires reviewing existing literature or to consult experts in the domain. To take a simple example, if you want to predict height of people, and you know from other previous studies that gender has an influence on height, it may be relevant to include gender as a feature in your model. $\endgroup$– J-J-JCommented Aug 12 at 10:47
-
2$\begingroup$ @Leilaali Perhaps you should focus first on fixing the part where "I have to delete the observations with NA values. So I have just 25 observations." Depending on the aim of your analysis (prediction? inference), better techniques than listwise deletion may be available. With 25 observations, every additional observation will help. $\endgroup$– Marjolein FokkemaCommented Aug 12 at 12:49
-
2$\begingroup$ A small number of observations means limited information, so you will need to live with the fact that training and testing will be of limited reliability. $\endgroup$– Christian HennigCommented Aug 12 at 13:06
|
Show 7 more comments
2 Answers
$\begingroup$
$\endgroup$
When undertaking an advanced analysis always ask yourself these questions:
- What is the simplest estimation task?
- Is my sample size adequate do to this task?
If the answer to the second questions is no, the analysis is futile. In your case the simplest task is to estimate the overall marginal mean of the response variable Y. Your sample size is far too small to do that. So it is far too small to try to estimate mean of Y conditional on a single pre-specified predictor, not to mention a 1000 candidate predictors.
In my experience with binary Y, the minimal sample size for split-sample validation to work (i.e., to be reliable / stable) is around 20,000. 100 repeats of 10-fold cross-validation can work, and you will be very disappointed with the results of this resampling validation on your small dataset.
answered Aug 12 at 13:50
$\begingroup$
$\endgroup$
This answer will not focus on whether variable selection in such a situation makes sense at all, or whether you should really try hard to make more observations available (imputation techniques for missing values may help).
Regarding the overfitting issue, however, in principle LOOCV may be as good as it gets to assess this, however you need to make sure that you don't have information leakage, meaning that you need to run the LOOCV in such a way that the final assessment of your prediction quality is really run on data which has not been used for model selection or dimension reduction at an earlier stage. One way to achieve this is nested cross-validation, i.e., you may want to assess the final prediction quality by an outer cross-validation that leaves one observation at a time out from the complete model selection procedure. If the latter also involves LOO-CV, you need to run this an on inner CV each time on the observations that do not include the one observation left out in the outer CV. (A similar thing can be done with bootstrap.)
As I said this is probably as good as it gets, but with the given number of observations the prediction error estimate may not be very precise anyway.
