3
$\begingroup$

I am attempting to analyze a small dataset using machine learning (SVM, binary problem). There are $103$ samples and $215$ variables (all variables are real numbers). Some of the variables (around half) are moderately/highly correlated with each other.

Knowing the empirical distribution of each variable, would it be somehow possible to generate more data by a simulation? I guess I should do it only for the training data. I am using nested cross-validation for picking parameters / estimating performance if it matters.

$\endgroup$
3
  • 1
    $\begingroup$ Generate more data for what purpose? $\endgroup$ May 23, 2015 at 19:31
  • 2
    $\begingroup$ If you already knew the joint distribution of Y and X, why would you want to fit a SVM? $\endgroup$
    – Adrian
    May 23, 2015 at 21:27
  • $\begingroup$ I don't think you need more data, really. SVMs have good regularization, you should be safe. Also something like Attribute Bagging is worth a try. $\endgroup$
    – Firebug
    Jun 23, 2016 at 22:38

3 Answers 3

2
$\begingroup$

You can ignore some of the correlated attributes and use Cross validation, instead of "generating" more data. Cross validation is a common approach in situations where you have more attributes than observations.

$\endgroup$
2
$\begingroup$

Similar to what stan0 said, I would use cross validation, but more specifically repeated k-fold Cross Validation. For a 'smaller' data set of 103 by 215, I would perhaps use 10x10 (10 repeats of 10 fold cross validation, so 100 model fits in total). If you're using R, I would highly suggest the caret package which is able to handle this for you using the trainControl() function.

Good resources for the underlying funciton calls and algorithum: https://topepo.github.io/caret/training.html https://topepo.github.io/caret/splitting.html

$\endgroup$
2
$\begingroup$

Knowing the empirical distribution of each variable, would it be somehow possible to generate more data by a simulation?

Yes, you can always generate more data, the question is of its quality! If you know the empirical distributions of the individual variables, this is not enough to generate more data. If you generated data on each of the variables independently, then you would end up with dataset where the features are unrelated to themselves and your target variable. So if your purpose is to check how does your model behaves with the data where the variables are unrelated to your target variables (i.e. how much is it prone to produce false positives), then this would help. Otherwise, such approach does not make sense. As others said, $k$-fold cross-validation and similar methods may help if size of your sample is limited (see also questions tagged as for other suggestions).

However, there are situations where you can produce artificial data for your model. It can be done, and often is done, in image processing where you can start with small dataset and then add noise to it (rotate images, add white noise, mix them with other pictures, change the backgrounds, make them blurry, change their sizes, resolution etc.). Unfortunately, this does not easily translate to other kinds of data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.