1
$\begingroup$

The group that I work with sent me 10 samples, 5 controls and 5 treatment. They asked me to try to use machine learning in this dataset, to see if it could separate the two groups.

I believe that this amount of data is very very low, and I'm quite sure that I can't get any reliable result from this. But, as a beginner in Machine Learning, I don't have the knowledge to explain to them why it is a complicated thing to do.

So, in order to understand better the problem here, my questions are:

  1. Is it is even possible to train a model using only 10 samples?

  2. If possible, what is the appropriate cross-validation approach to use in such small dataset?

  3. In order to explain to the group, why is it hard to get reliable results from small datasets?

Also, I'm trying to read as many articles as possible about ML, but the content is vast and I'm quite lost. Any recommendations of articles that discuss these problems that I pointed are very welcome.

$\endgroup$
3
  • 2
    $\begingroup$ 10 samples is...not great. How many features does each sample have? Your problem may be ill-posed, in which case it would need heavy regularization. If this is even worth doing, I'd suggest leave-one-out cross validation so that you're training on as many samples as possible. $\endgroup$ – Arya McCarthy Apr 21 at 22:38
  • $\begingroup$ 94 features. Thank you for your answer, I'll try the LOOCV. I'll also search for some materials for the heavy regularization step. $\endgroup$ – Lucas Lazari Apr 22 at 2:31
  • 1
    $\begingroup$ run something like logit and call it ML. that's what many do anyways. everyone's ML expert these days $\endgroup$ – Aksakal Apr 22 at 14:10
1
$\begingroup$

You can train a partition tree on ten data points even with an arbitrary amount of features and run LOOCV over that: If there is a very simple and nearly 100% reliable distinction (feature no 3 is 1 in group A and 2 in group B) than a tree could find that. If there is some complicated rule as distinction, you will not find that in n = 2 x 5.

Can you train a model? Yes, you can. The following will grow a randomForest on n = 2*5 with only two features. Is it sensible to train a randomForest on only two predictors? Who cares. On these obvious data it will compute an accuracy and correctly tell you, which of the two predictors contains the relevant information:

library(caret)

expl <- data.frame(group = gl(2, 5),
                   a = c(1, 2, 3, 4, 5, 10, 11, 12, 13, 14),
                   b = rnorm(10))

f <-train(group ~ ., 
          data = expl,
          method ="rf",
          trControl = trainControl(method = "LOOCV"),
          verbose = FALSE,
          importance = TRUE)
f

varImp(f)

Sometimes it is simple to get more observations, sometimes it's hard. If there is only very little information, you should first check, whether there is useful prior information for falling back to Bayes statistics. If not you need to decide whether it's best to say nothing at all based on weak evidence or whether it is worthwhile to use every little bit of information there is. Sometimes the latter is worthwhile even if unsatisfactory.

$\endgroup$
3
  • 2
    $\begingroup$ "If there is a very simple and nearly 100% reliable distinction...then a tree could find that" This is true, but suppose you did find such a pattern. The problem is that, with only 10 data points, it's hard to say whether this reflects a true/underlying pattern, or is just a random fluke where the few data points happened to line up that way. Cross validation won't necessarily cure this problem because it too needs enough data to produce a reliable estimate. That is, if 10 data points happened to line up nicely by chance, cross validation wouldn't give any indication that something is wrong $\endgroup$ – user20160 Apr 22 at 15:34
  • $\begingroup$ I see, thank you for both the explanations.. one more thing that I can't understand properly.. In this case, should I even bother to do hyperparameter tunning? I think that is a very wierd thing to do in such small dataset. $\endgroup$ – Lucas Lazari Apr 22 at 17:11
  • 1
    $\begingroup$ I tend to see it that way: If you find something you should call your finding a hypothesis. A hypothesis founded in data. Never a proof. If the client is going to take it as proof, don't do it in the first place. But some data are just too precious to not even look at them. Obviously don't overdo it with hyperparameter tuning. Don't stretch what Andrew Gelman calls "researcher's degrees of freedom". $\endgroup$ – Bernhard Apr 22 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.