# Tag Info

Accepted

### Can you be 93.75% confident from a random sample of only five from a population of 10 000?

Let's ignore the numbers for a bit. If we draw five observations from the population, the probability that all five observations are above the median is $\left({1\over 2}\right)^5 = 1/32 = 0.03125$, ...
• 40.7k
Accepted

### Does machine learning really need data-efficient algorithms?

You are not entirely wrong, often it will be a lot easier to collect more/better data to improve an algorithm than to squeeze minor improvements out of the algorithm. However, in practice, there are ...
• 34.3k

### Does machine learning really need data-efficient algorithms?

I work in retail forecasting. When you need to forecast tomorrow's demand for product X at store Y, you only have a limited amount of data available: possibly only the last two years' worth of sales ...
• 129k

### Can you be 93.75% confident from a random sample of only five from a population of 10 000?

Yes, this really works, under certain conditions, with a couple of caveats Random selection: You can't just ask any 5 people. It would need to be randomly selected from the population whose median ...
• 287k
Accepted

### Mother milk of 6 Corona-positive (COVID-19) women does not contain the virus - can we make a confidence statement about this?

There is the rule of three saying if a certain event did not occur in a sample with $n$ subjects, the interval from $0$ to $3/n$ is a 95% confidence interval for the rate of occurrences in the ...
• 81.7k

### Can I trust a significant result of a t-test if the sample size is small?

You should rarely trust any single significant result. You didn't say why you were using a one-tailed instead of a two-tailed test, so hopefully you have a good reason for doing so other than ...
• 536

### Can you be 93.75% confident from a random sample of only five from a population of 10 000?

The other answers have this exactly correct, but I'll explain why it seems so surprising. The trick is that the way the problem is posed hides the goalposts a little bit. We know we have a tiny sample ...
• 10.1k

### Train-validation-test split for small and unbalanced dataset?

However, my number of class 1 rows is so low that the way they get shuffled into validation or test set causes huge fluctuations in performance metrics. Here is another way of looking at things: your ...
• 129k
Accepted

### Neural network vs regression in a small sample

Neural networks, in vast majority of cases, need lots of data. If you have 20 observations, neural network is clearly a bad choice. With that small sample size, network would easily memorize the data ...
• 140k

### Is Random Forest suitable for very small data sets?

Random forest is basically bootstrap resampling and training decision trees on the samples, so the answer to your question needs to address those two. Bootstrap resampling is not a cure for small ...
• 140k
Accepted

### Can I trust a significant result of a t-test if the sample size is small?

In theory if all the assumptions of the t-test are true then there's no problem with a small sample size. In practice there are some not-quite-true assumptions which we can get away with for large ...
• 4,089

### Can I trust a significant result of a t-test if the sample size is small?

Imagine yourself to be in a situation where you're doing many similar tests, in a set of circumstances where some fraction of the nulls are true. Indeed, let's model it using a super-simple urn-type ...
• 287k

### Kolmogorov Smirnov Z vs Mann Whitney U small sample size n= 15?

If the original statement doesn't limit the conditions under which it applies pretty substantially, Field is just wrong on this. Responding to the quoted section: In effect, this means it does much ...
• 287k

### A huge gap between training and validation accuracy, confusion with the concept of Overfitting

Sounds like you are severely overfitting. Basically, you need to use a simpler model than the one you are currently using or collect (a lot) more data. Generally, the more data you have, the more ...
• 3,390
Accepted

### Which statistical test to use when there's no data of SD but just know the mean?

No, I do not believe there's any formal test that you can apply. (By the way, the usual term for such samples is pooled.) You are right that repeating the blot will not tell you more about the ...
• 2,260

### Train-validation-test split for small and unbalanced dataset?

A useful way to quantify the difficulty of the task is to compute the effective sample size as discussed here. Here the ESS is $3np(1-p)$ where $p=0.07$; ESS=19.5. With that amount of information ...
• 96.5k

### Does machine learning really need data-efficient algorithms?

While it is true that nowadays it is fairly easy to gather large piles of data, this doesn't mean that it is good data. The large datasets are usually gathered by scraping the resources freely ...
• 140k

### Does machine learning really need data-efficient algorithms?

Here are a couple thoughts to add to what has been posted so far. You might be interested in taking a look at the famous machine learning paper, Domingos, P. (2012). "A Few Useful Things to Know ...

### What is reasonable to do with small -tiny- datasets?. Dealing with rare diseases

Such datasets are too small for reliable biomarker development. The only hope is a proof-of-concept study, which is really a proof-of-signal-existence study where you have to put all your eggs in one ...
• 96.5k
Accepted

### Train-validation-test split for small and unbalanced dataset?

A few thoughts in addition to Stephan Kolassa's and Christian Henning's answers: As Christian says, nested cross validation is the only sensible splitting scheme in small sample size situations. It ...
• 39.4k

### How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples

One aspect that does not seem to have been raised in the other answers is that different tests (e.g., the t-test vs., the Mann-Whitney U test) test different things. The t-test is a test on the ...
• 129k
Accepted

### Ramifications of small + unbalanced group sizes, small number of groups for fixed & random effects models?

Mixed models are good at coping with unbalanced designs. This is one of their advantages compared to other approaches such as ANOVA-type models. So I would not worry about this. You mention small ...
• 63.9k

### Does machine learning really need data-efficient algorithms?

I was once asked to build a model that puts archeological artifacts into classes according to their manufactoring process. A big problem: for some classes, there were only four samples. And many ...
• 201
Accepted

### What are "poor finite sample properties"?

In the context of hypothesis tests, poor finite sample properties usually mean that the actual rejection rate of the test differs from the nominal one. Recall that the nominal one is the level at ...
• 34.3k

### Studies with small sample sizes

A comment made by "Josh" under this Andrew Gelman's blog post: https://statmodeling.stat.columbia.edu/2022/01/24/how-large-a-sample-size-does-he-actually-need-he-got-statistical-significance-...
• 2,593

### P-value adjustment for a single test with low sample size

No, for a single test you have a single p-value, thus no correction method is needed. Indeed, multiple comparisons problem arises when you perform many statistical tests or when you build many ...
• 12k

### Train-validation-test split for small and unbalanced dataset?

I agree with Stephan Kolassa generally. If you have only 7 observations from one class, the basis for hyperparameter selection and performance assessment is severely limited. Your data only carries a ...
• 26.9k
Accepted

### How to get the confidence interval of a Bernoulli trial if $\hat{p} = 0$?

The reason the usual "CLT" confidence interval becomes 0 is because when $p$ is very close to 0 or 1 (and the relative number of samples is low), the CLT becomes a bad approximation. This is because ...
• 14k