Skip to main content
22 votes

Train-validation-test split for small and unbalanced dataset?

However, my number of class 1 rows is so low that the way they get shuffled into validation or test set causes huge fluctuations in performance metrics. Here is another way of looking at things: your ...
Stephan Kolassa's user avatar
12 votes

Train-validation-test split for small and unbalanced dataset?

A useful way to quantify the difficulty of the task is to compute the effective sample size as discussed here. Here the ESS is $3np(1-p)$ where $p=0.07$; ESS=19.5. With that amount of information ...
Frank Harrell's user avatar
10 votes
Accepted

Train-validation-test split for small and unbalanced dataset?

A few thoughts in addition to Stephan Kolassa's and Christian Henning's answers: As Christian says, nested cross validation is the only sensible splitting scheme in small sample size situations. It ...
cbeleites unhappy with SX's user avatar
10 votes

Train-validation-test split for small and unbalanced dataset?

I agree with Stephan Kolassa generally. If you have only 7 observations from one class, the basis for hyperparameter selection and performance assessment is severely limited. Your data only carries a ...
Christian Hennig's user avatar
9 votes
Accepted

Comparing coefficients and confidence intervals when some categories have very few observations (logistic regression)

It's a good question, and you already have most of an answer: in a set-up like yours it is essential to look at the confidence intervals, not just the coefficient estimates. The machinery is intended ...
Nick Cox's user avatar
  • 58k
1 vote

Do you need to adjust the probability if you use the 'class_weight' parameter in LogisticRegression-sklearn?

You seem to be mixing up a few notions that often get mixed up in machine learning work. You start out by saying you want the probabilities and use a logistic regression to estimate them. Outstanding! ...
Dave's user avatar
  • 64.2k

Only top scored, non community-wiki answers of a minimum length are eligible