1
$\begingroup$

If I have a small data set (30 samples) am I more likely to obtain a statistically significant result? From my understanding if there are any relationships within the data set then they will be over represented in the data set? Is this true?

Thanks

$\endgroup$
5
$\begingroup$

You are asking a question about how likely something is. Questions about probabilities ALWAYS depend on a set of assumptions. Without giving way more context and specifying the assumptions you are making, your question cannot be answered.

If you assume there is no trend in the population (the actual trend line is horizontal), then sample size doesn't matter. There is a 5% chance of obtaining a P value less than 0.05. But if you assume there is an actual trend in the population, you are more likely to get a P value less than 0.05 with a larger sample than with a small one.

$\endgroup$
  • 1
    $\begingroup$ And if there is a trend that goes the anther way you expected, you are less likely to get a P value less than 0.05 with a larger sample than with a small one. $\endgroup$ – Peter Jan 28 '15 at 20:46
2
$\begingroup$

In statistical significance testing, we take a statement and see if there’s enough evidence in the data to believe that this statement is false. For example, consider the following statement: the average NBA game has 18,000 attendees. To test this hypothesis, we must collect data on the average number of attendees in a sample of NBA games.

There are three things that affect statistical significance – the first thing being the average in the data. For example, imagine two situations. In situation one, the data you collect has an average attendance of 19,000. In situation two, the data you collect has an average attendance of 18,100. With all else being equal, situation one shows more evidence in favor of rejecting the original statement and believing that the average NBA game has more than 18,000 attendees.

The second thing that affects statistical significance is sample size. If you have a sample of 10 NBA games and the average attendance is 18,100 then there’s likely not enough evidence to say that the average number of attendees at an NBA game is more than 18,000. However, if you have a sample of 10,000 NBA games and the average is 18,100 then there likely is enough evidence to say that the average number of attendees at an NBA game is more than 18,000 (it looks to be about 18,100 which is more than 18,000).

The third thing that affects statistical significance is variation in the data. Consider two situations. In situation one, you have a sample of 5 NBA games each with 19,000 attendees. In situation two, your sample of 5 NBA games has the following number of attendees: 17,000; 18,000; 19,000; 20,000; 21,000. In both cases the average in the sample is 19,000. However, in the second situation we have much more variation in the data and, consequentially, we have less confidence that the true average number of fans that attend an NBA game is 19,000 and do not have enough evidence to reject the original statement.

In statistical significance testing, the average of the data, the sample size, and the variation in the data all work together to determine statistical significance and none of these three things can be looked at in a vacuum.

$\endgroup$
2
$\begingroup$

No, it is exactly the opposite: your p-value gets smaller when your sample gets larger. See an example below.

First lets create a sample with some random noise and plot it:

set.seed(123)
n <- 10
x <- 1:n
y <- x + runif(n, -n, n)

enter image description here

And now lets run linear regression ten times, each time with sample being the same values but repeated $k$ times.

f <- function(k, x, y) {
   x <- rep(x, k)
   y <- rep(y, k)
   obj <- summary.lm(lm(y~x))
   pf(obj$fstatistic[1], obj$fstatistic[2], obj$fstatistic[3], lower.tail = FALSE)
}

pv <- NULL
for (k in 1:10)
   pv <- c(pv, f(k, x, y))

Lets plot p-values:

enter image description here

As you can see, nothing changes here, just the sample gets bigger and still, p-values get smaller and smaller. Check also this thread.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.