# How to handle generate more data in small data-sets for regression?

I have a logistic regression and decision tree classifiation model with a dataset with 70 data entries, yet only 16 responses have said yes for the response variable. I have 8 covariates.

1. How can i artificially generate more data? Does it make sense to?
2. How can i increase the accuracy with a small data-set? (Penalisations for ex?)

Instead of collecting more data, are there other ways to increase the data-set? For example, i have seen multiple imputation used for missing data, but can i do imputation when there is no missing data to generate more rows for all covariates? I also know of transfer learning in CNNs, but could this perhaps be used to pretrain a regression model?

It would be interesting to know of any statistical methods to generate more data, and perhaps methods to increase accuracy with small datasets, such as would lasso penalties be suitable? Thanks.

No. If you are sampled it, you would have the same data, just duplicated. If you used simulated data, your model would basically learn how to imitate the simulation, rather how to predict what was observed. You cannot generale something out of nothing. You need to either collect more data, or use a a very simple model (e.g. logistic regression with the single feature). If you have strong prior knowledge, you could use it within Bayesian model to augment the data with the prior information.

• But i would need strong prior knowledge of each variable to augment the data right? So i assume it is very difficult to do this.
– jojo
Mar 26, 2022 at 8:55
• @jojo that's correct
– Tim
Mar 26, 2022 at 9:38

To extend Tim's answer (+1) a bit, if you don't have strong prior information, ridge regression and LASSO can be thought of as "Bayes estimates with different priors"; see Section 3.4.3 of The Elements of Statistical Learning.

With your limited data, however, LASSO with cross-validated penalty selection would probably only return 1 or 2 predictors with non-zero coefficients. Furthermore, the particular selected predictors would probably differ if you repeated the process on multiple bootstrapped samples. Ridge regression would lead to highly penalized coefficients for all predictors.

If one predictor is of major interest and you just want to adjust somewhat for the others, you could use this approach, keeping that predictor unpenalized while applying a ridge penalty to the others.

Transfer learning can't handle small sample. U could try random undersampling, oversampling or SMOTE to balance the response variable. Here is the best book for this kinda situation:

https://www.amazon.com/Learning-Imbalanced-Data-Alberto-Fern%C3%A1ndez/dp/3319980734