0
$\begingroup$

Are there a minimum number of observations for ML techniques (classification, regression) in psychology/cognitive neuroscience? In particular for training and test datasets? I found this article for sample size in biology(https://arxiv.org/abs/1211.1323) but any for regression ML

I have a 25 obs data with 38 predictors on a continuous dependent variable, I was thinking to use PLS regression as some predictors are highly correlated and predictors > N. How can I split the dataset? Any suggestions?

Thanks

$\endgroup$
1
  • 1
    $\begingroup$ There are no such rules in any field. It depends entirely on your data. The only rule is get more data. $\endgroup$ – user2974951 Nov 7 '19 at 11:23
1
$\begingroup$

That (our) paper applies independent of the field, but wrt. to test sample size it is only about figures of merit that are proportions of test cases (sensitivity, specificity, ...).

You'll find that these figures of merit are not recommended for many situations, among other reasons (they are no proper scoring rules) because they have high variance. This high variance is directly related to our calculations, and we exploit the fact that upper and lower boundaries can be calculated beforehand. Still, in practice, you may find that you need to work with these figures of merit.

I'm currently working at extending this to regression and other figures of merit. There, the variance uncertainty due to the limited number of test cases cannot be calculated in advance as for the proportions, but it can be measured from (internal) validation experiments. Thus, estimates will need to be done on the basis of preliminary experiments.

If you try to calculate necessary training sample size this way, you'll need to keep in mind that extrapolating (as in predicting how many further samples you need to run) is extremely uncertain as there can be interactions between this uncertainty on the test results and the model quality which is also influenced by sample size (and model complexity usually changes as well). There are some papers out doing this (we cite a few), but at least for me the result that we may get where we want to be with something between 200 and 10000 further samples is not of much practical use...


For training of linear models, there are rules of thumb that will likely yield a stable model. The important parameter here is the ratio between training sample size and number of features (≈ model complexity).

In contrast to testing, this means that you can always adapt the model to the too-small sample size that is available: the resulting model may not be good, but it will be as good as possible given sample size, model type and problem. With testing, there is no similar trade-off that you can employ: testing depends on the absolute number of available test cases.


The important bottomline of that paper is that in small-n-high-p small sample size situations such as yours the bottleneck is (internal) validation rather than training. As a sneak-preview to ongoing work, I have similar findings for some regression model where I looked at RMSE though I cannot say whether the internal testing is always a bottleneck here, too. But I can say that one should not be surprised to find this the bottleneck.

Also testing being the bottleneck becomes important for training if such test results are used for data-driven model optimization (hyperparameter tuning).


How can I split the data set

With such extremely small data sets, the best you can do is

  • refrain from data driven model optimization: use models where you can employ external knowledge (professional experience) to determine the necessary hyperparameters.
  • Use resampling validation such as out-of-bootstrap (or its variants) or cross valiation (preferrably repeated/iterated to check model stability).
    A single split (train/test) is a waste that you cannot afford with so few samples.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.