Questions tagged [small-sample]
Refers to statistical complications or problems due to having few data. If your question is about a small sample relative to the number of variables, please use the [underdetermined] tag instead.
712
questions
143
votes
9
answers
122k
views
How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples
Certain hypotheses can be tested using Student's t-test (maybe using Welch's correction for unequal variances in the two-sample case), or by a non-parametric test like the Wilcoxon paired signed rank ...
- 23.8k
96
votes
4
answers
66k
views
Can bootstrap be seen as a "cure" for the small sample size?
This question has been triggered by something I read in this graduate-level statistics textbook and also (independently) heard during this presentation at a statistical seminar. In both cases, the ...
- 2,754
71
votes
6
answers
178k
views
Warning in R - Chi-squared approximation may be incorrect
I have data showing fire fighter entrance exam results. I am testing the hypothesis that exam results and ethnicity are not mutually independent. To test this, I ran a Pearson chi-square test in R. ...
- 825
58
votes
6
answers
71k
views
Best method for short time-series
I have a question related to modeling short time-series. It is not a question if to model them, but how. What method would you recommend for modeling (very) short time-series (say of length $T \leq 20$...
- 140k
50
votes
5
answers
11k
views
What can we say about population mean from a sample size of 1?
I am wondering what we can say, if anything, about the population mean, $\mu$ when all I have is one measurement, $y_1$ (sample size of 1). Obviously, we'd love to have more measurements, but we can'...
- 525
33
votes
3
answers
7k
views
Can you be 93.75% confident from a random sample of only five from a population of 10 000?
I recently read this passage from a website and I just can't work out the math.
Overall, it says you can be 93.75% confident of having the true median parameter within an interval, obtained from a ...
- 433
30
votes
4
answers
16k
views
How to perform Student's t-test having only sample size, sample average and population average are known?
Student's $t$-test requires the sample standard deviation $s$. However, how do I compute for $s$ when only the sample size and sample average are known?
For example, if sample size is $49$ and sample ...
- 443
30
votes
2
answers
3k
views
How to cope with exploratory data analysis and data dredging in small-sample studies?
Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
- 54.1k
29
votes
1
answer
27k
views
Using bootstrap under H0 to perform a test for the difference of two means: replacement within the groups or within the pooled sample
Suppose that I have a data with two independent groups:
...
- 407
24
votes
7
answers
19k
views
Appropriate normality tests for small samples
So far, I've been using the Shapiro-Wilk statistic in order to test normality assumptions in small samples.
Could you please recommend another technique?
- 2,211
24
votes
2
answers
4k
views
Mother milk of 6 Corona-positive (COVID-19) women does not contain the virus - can we make a confidence statement about this?
I am asking this question because I believe it would be great if the statistics community could make a contribution to solving this serious puzzle until more evidence is available.
The UK Royal ...
- 6,634
23
votes
2
answers
48k
views
Can a small sample size cause type 1 error?
I've learnt that small sample size may lead to insufficient power and type 2 error. However, I have the feeling that small samples just may be generally unreliable and may lead to any kind of result ...
- 2,387
23
votes
2
answers
7k
views
Topic stability in topic models
I am working on a project where I want to extract some information about the content of a series of open-ended essays. In this particular project, 148 people wrote essays about a hypothetical student ...
- 3,272
22
votes
11
answers
5k
views
Does machine learning really need data-efficient algorithms?
Deep learning methods are often said to be very data-inefficient, requiring 100-1000 examples per class, where a human needs 1-2 to reach comparable classification accuracy.
However, modern datasets ...
- 1,337
22
votes
2
answers
48k
views
Is Random Forest suitable for very small data sets?
I have data set comprising 24 rows of monthly data. The features are GDP, airport arrivals, month, and a few others. The dependent variable is number of visitors to a popular tourism destination. ...
- 321
22
votes
2
answers
8k
views
Mean(scores) vs Score(concatenation) in cross validation
TLDR:
My dataset is pretty small (120) samples. While doing 10-fold cross validation, should I:
Collect the outputs from each test fold, concatenate them into a vector, and then compute the error on ...
- 885
19
votes
1
answer
4k
views
$t$-tests vs $z$-tests?
I'm trying to figure out exactly what the difference is between $t$-tests and $z$-tests.
As far as I can tell, for both classes of tests one uses the same test statistic, something of the form
$$\...
- 1,967
19
votes
1
answer
808
views
Is bootstrap problematic in small samples?
In "3 Things That Bother Me" (1988), Ed Leamer writes:
Bootstrap estimates of standard errors are based on the assumption that the observed sample is the same as the true distribution, ...
- 68.9k
18
votes
8
answers
75k
views
Minimum sample size for unpaired t-test
Is there a "rule" to determine the minimum sample size required for a t-test to be valid?
For example, a comparison needs to be performed between the means of 2 populations. There are 7 data points ...
- 183
17
votes
6
answers
7k
views
Can I trust a significant result of a t-test if the sample size is small?
If my one sided t-test result is significant but the sample size is small (e.g. below 20 or so), can I still trust this result? If not, how should I deal and/or interpret this result?
- 516
16
votes
3
answers
6k
views
Bootstrap: the issue of overfitting
Suppose one performs the so-called non-parametric bootstrap by drawing $B$ samples of size $n$ each from the original $n$ observations with replacement. I believe this procedure is equivalent to ...
- 2,754
15
votes
4
answers
1k
views
Train-validation-test split for small and unbalanced dataset?
I have a dataset of around 100 rows, each with around 400 features. 93 of them are class 0, and 7 are class 1.
I want to be able to split my 100 examples into a train set, a validation set, and a test ...
- 153
15
votes
1
answer
5k
views
Halton sequence vs Sobol' sequence?
From an answer in a previous question, I was directed toward the Halton sequence, for creating a set of vectors that covered a uniform sample space fairly evenly. But the wikipedia page mentions that ...
- 5,501
14
votes
7
answers
9k
views
Are short time series worth modelling?
Here is some context. I am interested in determining how two environmental variables (temperature, nutrient levels) impact the mean value of a response variable over a 11 year period. Within each year,...
- 363
14
votes
0
answers
382
views
How to construct confidence limits based on small stratified samples of finite populations?
Imagine a business wishes to audit its transactions. It has a database summarizing the transactions, which constitute a sampling frame for the population. It would be time-consuming and expensive to ...
- 330k
13
votes
4
answers
4k
views
Graphing small samples
I have a small data set of 14 separate times to complete a task. However I am having difficulty finding an appropriate graph to use to graph the data. If the sample was larger I would use a box plot ...
- 135
13
votes
1
answer
45k
views
Regression with very small sample size
I want to run a regression with 4 to 5 explanatory variables, but I have only 15 observations. Not being able to assume these variables are normally distributed, is there a non-parametric or any other ...
- 131
13
votes
0
answers
379
views
Is autocorrelation not worth addressing with small N?
Consider a simple regression context in which there is a small set of response values, $Y$, and corresponding dates, $X$. (For simplicity, we can assume the dates are equally spaced.) We would like ...
12
votes
2
answers
4k
views
Is randomization reliable with small samples?
Jerome Cornfield has written:
One of the finest fruits of the Fisherian revolution was the idea of
randomization, and statisticians who agree on few other things have at
least agreed on this. ...
- 2,001
12
votes
3
answers
3k
views
Limits to tree-based ensemble methods in small n, large p problems?
Tree-based ensemble methods such as Random Forest, and subsequent derivatives (e.g., conditional forest), all purport to be useful in so-called "small n, large p" problems, for identifying relative ...
- 759
12
votes
1
answer
1k
views
ANOVA: testing assumption of normality for many groups with few samples per group
Assume the following situation:
we have a large number (e.g. 20) with small group sized (e.g. n = 3). I noticed that if I generate values from the uniform distribution, the residuals will look ...
- 7,279
11
votes
1
answer
5k
views
Oversampling: whole set or training set
I have a rather small dataset of 4 000 points (140 features) to feed to a NN binary classifier. The problem is only ~700 of them represent the second class. Is it more common to resample the whole ...
- 537
10
votes
3
answers
2k
views
Studies with small sample sizes
I'm asking myself the question of why studies with small sample sizes are not as convincing as those with larger sample sizes, and when this becomes a statistical issue. A complaint I've heard a lot ...
- 183
10
votes
1
answer
4k
views
Correction for multiple testing on a modest number of tests (10-20) with FDR?
False Discovery Rate (Benjamini-Hochberg) is typically used on 'Big Data', such as genetic studies using 100s of tests. But can it also be used on smaller numbers of tests? For example, looking at the ...
- 143
10
votes
1
answer
333
views
What are "poor finite sample properties"?
In MacKinlay’s (1997) well-known article about event study methodologies, he states that there are two ways event clustering can be accommodated in event studies. Event clustering means that event ...
- 103
9
votes
2
answers
10k
views
Is the sampling distribution for small samples of a normal population normal or t distributed? [closed]
If I know that the population is normally distributed, and then take small samples from this population, is it more correct to claim that the sampling distribution is normal or instead follows the t ...
- 91
9
votes
2
answers
17k
views
Good classifiers for small training sets
What are good classification methods in the case of continuous independent variables (features) and a small training set (particularly where the number of training examples is approximately equal to ...
- 581
9
votes
3
answers
4k
views
Dimension reduction techniques for very small sample sizes
I have 21 socio-economic and attitudinal macro-level variables (such as percentage of mothers aged 24-54 not employed, percentage of children aged 3-5 in nursery schools and so on). I also have data ...
- 91
9
votes
3
answers
413
views
Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?
In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
- 641
9
votes
2
answers
7k
views
Within-subject covariance estimation in a small sample
My sample has four subjects whose response variable is each measured four times. I assume the measurements within each subject is normally distributed $\mathcal{N}_4(\mu,\Sigma)$.
Is there any good ...
- 461
8
votes
5
answers
968
views
Correlation for Small Dataset?
I have an $x$ and a $y$ that I would like to find the correlation of to learn more about their relationship. Unfortunately, I only have $10$ points. Can I in good faith use the Pearson correlation ...
- 81
8
votes
3
answers
2k
views
Regression for really small data with high degree of multicollinearity and outliers
I'm working on a promotional response analysis. I have a really small real world dataset with 25 observations and 15 variables. The variables have a high degree of multicollinearity and some have ...
- 8,515
8
votes
1
answer
3k
views
Kolmogorov Smirnov Z vs Mann Whitney U small sample size n= 15?
I have a small sample size of 15. I want to see if there is a difference in the nutrient intakes between two independent variables, group 1 n = 11, group 2 n= 4. The data is not normally distributed. ...
- 81
8
votes
1
answer
6k
views
Is PCA appropriate when $n<p$?
This question is an extension of one I asked a few weeks back:
Minimum sample size for PCA or FA when the main goal is to estimate only few components?
I will restate that I am interested in the use ...
- 1,581
8
votes
1
answer
2k
views
How to get the confidence interval of a Bernoulli trial if $\hat{p} = 0$?
I know the standard formula for the Bernoulli CI is:
$$\hat{p}\pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$
If $\hat{p} = \frac{m}{n}$ how do I estimate the confidence interval when$\ n$ ...
- 183
8
votes
4
answers
19k
views
Machine learning models for regression on small data sets
What are the "best" models to be used for simple regression of 1 numerical variable using only a small data set of e.g. 250 samples and up to 10 features?
I understand that the data set is super ...
- 727
8
votes
1
answer
12k
views
Is significance of the p-value reliable with extremely small sample sizes? [duplicate]
As I'm completely new to statistics and clinical trials, I would ask you for help and clarification.
Let's assume we have 2 really small samples (each of n = 2, or so) derived from two normally ...
- 83
8
votes
1
answer
6k
views
Multiple regression with small data sets
I have a dataset of project case studies for a new type of research method for Government agencies to support decision making activities. My task is to develop an estimation method based on past ...
- 131
8
votes
1
answer
2k
views
Elastic net arbitrary alpha selection
I'm trying to solve a prediction problem given the following constraints:
I need an interpretable model to be used for experimental validation
I need a model that performs feature selection to reduce ...
- 91
8
votes
1
answer
2k
views
Comparing unbalanced groups with ANOVA/Kruskal-Wallis when one group has only 1 observation
I would like to compare a continuous variable across 5 health status groups, but one of the groups has only 1 observation. Would an ANOVA/Kruskal-Wallis be valid? What can I do about the group with ...
- 524