1
$\begingroup$

We are taught to split our data set into three parts but that doesn't align with what I learned in data analysis: power analysis. Do we really need thirty percent or twenty percent of data for the testing and validation? According to the power analysis, the sample size would be much fewer. I learned that 601(with 95% confidence level and margin of error smaller than 0.04) for the binary classification and 510(with alpha as 0.05 and confidence interval 95%) for the multi-class classification would be sufficient.

We only consider classification tasks since numerical data would be more complicated and most of our problems are classification problems.

Can we randomly choose minimum cases from the whole data set calculated by the minimum sample size in power analysis for both the validation set and test set in machine learning(for instance in NLP tasks) and use all other data as the training data set? I know more data would lead to a higher power, but if we are not that demanding would a minimum sample size practical?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Statistical power is not usually the issue when doing prediction, but a loss of statistical power is related to the precision of predictions and to the volatility of the found form of the predictive model. Data splitting harms all of these. That's why 100 repeats of 10-fold cross-validation, or the bootstrap, are recommended resampling procedures to do away with the need for holding back data. If you insist on data splitting, instead of power, find out which data splits would give you the lowest mean squared error of the predictive accuracy measure of interest. And make sure the accuracy measure is a proper scoring rule. For example if $Y$ is Gaussian conditional on $X$, mean squared error is an excellent scoring rule, and you should seek sample sizes that minimize the mean squared error of the mean squared error.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks very much for your answer, professor. I thought we are concerning most about the Y that are proportions. If we use 10-fold cross-validation or bootstrap would be a minimum sample size appropriate? $\endgroup$
    – Lerner Zhang
    Commented Jan 31, 2021 at 12:37
  • 1
    $\begingroup$ One 10-fold cross-validation is insufficient. You need to repeat that 50 or 100 times. But think about resampling methods this way: Whatever total sample size you have, you have to live within the confines of that. But use all samples for both model development and model validation so you don't have to lower the sample size any further. Details in hbiostat.org/rms course notes and book. $\endgroup$
    – Frank Harrell
    Commented Jan 31, 2021 at 13:13
  • $\begingroup$ I attempted to write an answer to this question: Sample size for the evaluation of Deep Learning Models, and could you please provide some feedback/suggestions, either negative or positive? Thanks in advance. $\endgroup$
    – Lerner Zhang
    Commented Feb 19, 2021 at 3:11
  • 1
    $\begingroup$ The estimates there are in the ballpark, assuming you are using resampling methods so that you don't need to hold back any data. But note that that post is pushing us to use poor accuracy scoring rules. $\endgroup$
    – Frank Harrell
    Commented Feb 19, 2021 at 12:14
  • $\begingroup$ Ok, let me learn more to comprehend that. $\endgroup$
    – Lerner Zhang
    Commented Feb 19, 2021 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.