70
votes
Accepted
Can you be 93.75% confident from a random sample of only five from a population of 10 000?
Let's ignore the numbers for a bit. If we draw five observations from the population, the probability that all five observations are above the median is $\left({1\over 2}\right)^5 = 1/32 = 0.03125$, ...
- 40.7k
37
votes
Accepted
Does machine learning really need data-efficient algorithms?
You are not entirely wrong, often it will be a lot easier to collect more/better data to improve an algorithm than to squeeze minor improvements out of the algorithm.
However, in practice, there are ...
- 34.3k
28
votes
Does machine learning really need data-efficient algorithms?
I work in retail forecasting. When you need to forecast tomorrow's demand for product X at store Y, you only have a limited amount of data available: possibly only the last two years' worth of sales ...
- 129k
26
votes
Can you be 93.75% confident from a random sample of only five from a population of 10 000?
Yes, this really works, under certain conditions, with a couple of caveats
Random selection: You can't just ask any 5 people. It would need to be randomly selected from the population whose median ...
- 287k
23
votes
Accepted
Mother milk of 6 Corona-positive (COVID-19) women does not contain the virus - can we make a confidence statement about this?
There is the rule of three saying
if a certain event did not occur in a sample with $n$ subjects, the interval from $0$ to $3/n$ is a 95% confidence interval for the rate of occurrences in the ...
- 81.7k
22
votes
Can I trust a significant result of a t-test if the sample size is small?
You should rarely trust any single significant result. You didn't say why you were using a one-tailed instead of a two-tailed test, so hopefully you have a good reason for doing so other than ...
- 536
22
votes
Can you be 93.75% confident from a random sample of only five from a population of 10 000?
The other answers have this exactly correct, but I'll explain why it seems so surprising. The trick is that the way the problem is posed hides the goalposts a little bit. We know we have a tiny sample ...
- 10.1k
22
votes
Train-validation-test split for small and unbalanced dataset?
However, my number of class 1 rows is so low that the way they get shuffled into validation or test set causes huge fluctuations in performance metrics.
Here is another way of looking at things: your ...
- 129k
19
votes
Accepted
Neural network vs regression in a small sample
Neural networks, in vast majority of cases, need lots of data. If you have 20 observations, neural network is clearly a bad choice. With that small sample size, network would easily memorize the data ...
- 140k
17
votes
Is Random Forest suitable for very small data sets?
Random forest is basically bootstrap resampling and training decision trees on the samples, so the answer to your question needs to address those two.
Bootstrap resampling is not a cure for small ...
- 140k
17
votes
Accepted
Can I trust a significant result of a t-test if the sample size is small?
In theory if all the assumptions of the t-test are true then there's no problem with a small sample size.
In practice there are some not-quite-true assumptions which we can get away with for large ...
- 4,089
16
votes
Can I trust a significant result of a t-test if the sample size is small?
Imagine yourself to be in a situation where you're doing many similar tests, in a set of circumstances where some fraction of the nulls are true.
Indeed, let's model it using a super-simple urn-type ...
- 287k
14
votes
Kolmogorov Smirnov Z vs Mann Whitney U small sample size n= 15?
If the original statement doesn't limit the conditions under which it applies pretty substantially, Field is just wrong on this.
Responding to the quoted section:
In effect, this means it does much ...
- 287k
13
votes
A huge gap between training and validation accuracy, confusion with the concept of Overfitting
Sounds like you are severely overfitting. Basically, you need to use a simpler model than the one you are currently using or collect (a lot) more data. Generally, the more data you have, the more ...
- 3,390
12
votes
Accepted
Which statistical test to use when there's no data of SD but just know the mean?
No, I do not believe there's any formal test that you can apply. (By the way, the usual term for such samples is pooled.)
You are right that repeating the blot will not tell you more about the ...
- 2,260
12
votes
Train-validation-test split for small and unbalanced dataset?
A useful way to quantify the difficulty of the task is to compute the effective sample size as discussed here. Here the ESS is $3np(1-p)$ where $p=0.07$; ESS=19.5. With that amount of information ...
- 96.5k
11
votes
Does machine learning really need data-efficient algorithms?
While it is true that nowadays it is fairly easy to gather large piles of data, this doesn't mean that it is good data. The large datasets are usually gathered by scraping the resources freely ...
- 140k
11
votes
Does machine learning really need data-efficient algorithms?
Here are a couple thoughts to add to what has been posted so far.
You might be interested in taking a look at the famous machine learning paper, Domingos, P. (2012). "A Few Useful Things to Know ...
11
votes
What is reasonable to do with small -tiny- datasets?. Dealing with rare diseases
Such datasets are too small for reliable biomarker development. The only hope is a proof-of-concept study, which is really a proof-of-signal-existence study where you have to put all your eggs in one ...
- 96.5k
11
votes
Accepted
Train-validation-test split for small and unbalanced dataset?
A few thoughts in addition to Stephan Kolassa's and Christian Henning's answers:
As Christian says, nested cross validation is the only sensible splitting scheme in small sample size situations. It ...
- 39.4k
10
votes
How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples
One aspect that does not seem to have been raised in the other answers is that different tests (e.g., the t-test vs., the Mann-Whitney U test) test different things.
The t-test is a test on the ...
- 129k
10
votes
Accepted
Ramifications of small + unbalanced group sizes, small number of groups for fixed & random effects models?
Mixed models are good at coping with unbalanced designs. This is one of their advantages compared to other approaches such as ANOVA-type models. So I would not worry about this.
You mention small ...
- 63.9k
10
votes
Does machine learning really need data-efficient algorithms?
I was once asked to build a model that puts archeological artifacts into classes according to their manufactoring process. A big problem: for some classes, there were only four samples. And many ...
- 201
10
votes
Accepted
What are "poor finite sample properties"?
In the context of hypothesis tests, poor finite sample properties usually mean that the actual rejection rate of the test differs from the nominal one.
Recall that the nominal one is the level at ...
- 34.3k
10
votes
Studies with small sample sizes
A comment made by "Josh" under this Andrew Gelman's blog post:
https://statmodeling.stat.columbia.edu/2022/01/24/how-large-a-sample-size-does-he-actually-need-he-got-statistical-significance-...
- 2,593
10
votes
P-value adjustment for a single test with low sample size
No, for a single test you have a single p-value, thus no correction method is needed. Indeed, multiple comparisons problem arises when you perform many statistical tests or when you build many ...
- 12k
10
votes
Train-validation-test split for small and unbalanced dataset?
I agree with Stephan Kolassa generally. If you have only 7 observations from one class, the basis for hyperparameter selection and performance assessment is severely limited. Your data only carries a ...
- 26.9k
9
votes
Accepted
How to get the confidence interval of a Bernoulli trial if $\hat{p} = 0$?
The reason the usual "CLT" confidence interval becomes 0 is because when $p$ is very close to 0 or 1 (and the relative number of samples is low), the CLT becomes a bad approximation. This is because ...
- 14k
9
votes
Can I trust a significant result of a t-test if the sample size is small?
Some of Gosset's original work (aka Student), for which he developed the t test, involved yeast samples of n=4 and 5. The test was specifically designed for very small samples. Otherwise, the normal ...
- 14.5k
9
votes
t test with drastically different sample sizes
The Welch approximation t-test is designed to do the same thing as an independent samples t-test, but without relying on the assumption that the variances are equal. It is readily available in most ...
- 2,225
Only top scored, non community-wiki answers of a minimum length are eligible