Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a small data set with about 45 observations(rows) and 60 variable(column). the data very small and it was difficult to create a responsible regression tree model with this data.
I tried to repeat the rows 10 times, and I surprised the the model quality is improved.
I'm not sure if this way is wrong and cause over fitting. Any advice or comment on this? Thanks
With 'oversampling', what you are doing is effectively changing the mutual weights of the samples. The mutual weight here is proportional to the number of times an observation is re-sampled.
The regression model you fit will be biased towards the observations with higher mutual weight (given that you haven't modified the loss function definition as well).
If you did sufficient cross-validation with purely randomized 'oversampling' you will find that model quality is in fact not improved. I suspect the reason you have arrived at the erroneous conclusion of improved model quality is that the cross-validation i.e. train-test split is done wrong. Actually, with 'oversampling' and proper cross-validation (training observations do not occur in test observations), your model quality should actually become worse. This is due to increased weight of some training samples and therefore increased bias in training data.
In conclusion, you are correct in your intuition that 'oversampling' is causing over-fitting. However, improvement in model quality is exact opposite of over-fitting, so that part is wrong and you need to check your train-test split in cross-validation. See: http://scikit-learn.org/stable/modules/cross_validation.html
--
Beyond the question asked, a tip would be to use feature selection. You can use a simple PCA step to find the principal components. It is unlikely that all 60 variables (or attributes) are equally important (typically not the case for real world data). You can retrain your regression model on just the principal components, or just the most important variables (columns) and see if that is more helpful to your overall objectives.
I disagree with the other comment-or on "oversampling". There is such a thing as "over-sampling" but it is a relative term and more related to design of experiment than simply fitting data like you are doing here. Once you have the data, we call it "up-sampling" when you repeat observations/measurements -- but this is another advanced trick akin to re-weighting your data and should only be done when you know exactly what you're doing and why (for instance, you want to bias your model purposefully).
That being said, your dealing with high dimensional data and is a classic problem in machine learning. You need to look at techniques that can handle high-dimensional data properly...primarily using cross validation of some kind (hence the forum name -- Cross-Validated :-)). I advise against PCA, it's not a very useful method in practice -- you're better off going straight for the goal using all features but using regularization via a validation set. This can include LASSO/Ridge regression, PLS, ANN, or others depending on your data and what you desire.
Note that LASSO/Ridge have issues when you have more features than observations which is one of the reasons PLS is used (so you may want to investigate PLS first).
I would suggest taking a look at imbalanced learn, SMOTE, ADASYN and RandomOverSampling could be considered. The models use KNN to to produce oversampled data. However, for highly imbalanced datasets overfitting is hard to avoid. Repeating the same row is not recommended.
