11
$\begingroup$

How do you perform hypothesis tests with big data? I wrote the following MATLAB script to emphasize my confusion. All it does is generate two random series, and run a simple linear regression of one variable on the other. It performs this regression several times using different random values and reports averages. What tends to happen is as I increase the sample size, p-values on average get very small.

I know that because the power of a test increases with sample size, given a large enough sample, p-values will become small enough, even with random data, to reject any hypothesis test. I asked around and some people said that with 'Big Data' its more important to look at the effect size, ie. whether the test is significant AND has an effect large enough for us to care about. This is because in large sample sizes the p-values will pick up of very small differences, like it is explained here.

However, the effect size can determined by scaling of the data. Below I scale the explanatory variable to a small enough magnitude that given a large enough sample size, it has large significant effect on the dependent variable.

So I'm wondering, how do we gain any insight from Big Data if these problems exist?

%make average
%decide from how many values to make average
obs_inside_average = 100;

%make average counter
average_count = 1;

for average_i = 1:obs_inside_average,






%do regression loop
%number of observations
n = 1000;

%first independent variable (constant term)
x(1:10,1) = 1; 

%create dependent variable and the one regressor
for i = 1:10,

    y(i,1) = 100 + 100*rand();

    x(i,2) = 0.1*rand();

end





%calculate coefficients
beta = (x'*x)\x'*y;

%calculate residuals
u = y - x*beta;

%calcuatate sum of squares residuals
s_2 = (n-2)\u'*u;

%calculate t-statistics
design = s_2*inv(x'*x);

%calculate standard errors
stn_err = [sqrt(design(1,1));sqrt(design(2,2))];

%calculate t-statistics
t_stat(1,1) = sqrt(design(1,1))\(beta(1,1) - 0);
t_stat(2,1) = sqrt(design(2,2))\(beta(2,1) - 0);

%calculate p-statistics
p_val(1,1) = 2*(1 - tcdf(abs(t_stat(1,1)), n-2));
p_val(2,1) = 2*(1 - tcdf(abs(t_stat(2,1)), n-2));






%save first beta to data column 1
data(average_i,1) = beta(1,1);

%save second beta to data column 2
data(average_i,2) = beta(2,1);

%save first s.e. to data column 3
data(average_i,3) = stn_err(1,1);

%save second s.e. to data column 4
data(average_i,4) = stn_err(2,1);

%save first t-stat to data column 5
data(average_i,5) = t_stat(1,1);

%save second t-stat to data column 6
data(average_i,6) = t_stat(2,1);

%save first p-val to data column 7
data(average_i,7) = p_val(1,1);

%save second p-val to data column 8
data(average_i,8) = p_val(2,1);

end

%calculate first and second beta average
b1_average = mean(data(:,1));
b2_average = mean(data(:,2));

beta = [b1_average;b2_average];

%calculate first and second s.e. average
se1_average = mean(data(:,3));
se2_average = mean(data(:,4));

stn_err = [se1_average;se2_average];

%calculate first and second t-stat average
t1_average = mean(data(:,5));
t2_average = mean(data(:,6));

t_stat = [t1_average;t2_average];

%calculate first and second p-val average
p1_average = mean(data(:,7));
p2_average = mean(data(:,8));

p_val = [p1_average;p2_average];

beta
stn_err
t_stat
p_val
$\endgroup$
  • $\begingroup$ Hypothesis testing is about rejecting null models. With more data you can reject "bigger null models", e.g. by having more covariates or testing multiple hypotheses. $\endgroup$ – momeara Aug 13 '13 at 19:43
  • $\begingroup$ Also see stats.stackexchange.com/questions/2492/… $\endgroup$ – nico Aug 13 '13 at 20:24
  • 2
    $\begingroup$ The elephant in the room is the representativeness of the "big data." Many huge datasets collected on the Internet are (at best) samples of convenience; there are hidden but well-known dangers lurking in the effort to generalize from the sample to a larger population or ongoing process. $\endgroup$ – whuber Aug 13 '13 at 21:17
  • 1
    $\begingroup$ "Some people said that with 'Big Data' its more important to look at the effect size." With 'Small Data' it is important to look at the effect size too. $\endgroup$ – Ari B. Friedman Mar 4 '14 at 1:14
11
$\begingroup$

As Peter suggested, I think one of the important things in the era of "Big Data" is to put even less emphasis on p-values, and more on an estimation of the magnitude of effect.

Some of my own work struggles with this in ways I think are even more insidious than with Big Data - for stochastic computational models, your power is entirely a function of patience and computing resources. It's an artificial construct.

So turn back to the effect estimate. Even if its significant, does a 0.0001% increase in something matter in the real world?

I've also been playing around with reversing some of the ideas behind reporting study power. Instead of reporting the power your study had to detect the observed effect, reporting the minimum effect size the study was powered to find. That way the reader can know if significance was essentially guaranteed.

$\endgroup$
7
$\begingroup$

The insight that you desire is going to come from confidence intervals, not as much from p-values. With a very large sample size you are going to get a very precise confidence intervals, provided your statistical assumptions are correct.

$\endgroup$
  • $\begingroup$ Thanks Mike. Are you saying in these situations, examination of the confidence intervals will show that they are so wide that we should not really trust the exact value of our estimates? $\endgroup$ – JoeDanger Aug 13 '13 at 20:58
  • $\begingroup$ What is interesting is how intuitively, the question was phrased as a problem for large data (where hypothesis tests informing us how unlikely something is to be 0 are rather pointless), and not as a problem for small data (where parameter estimates are very imprecise and often, all one could say is how unlikely the parameter is to be precisely 0). $\endgroup$ – jona Aug 14 '13 at 8:47
6
$\begingroup$

It is important to look at effect size regardless of whether the data are big or small.

With purely random data, you should get a significant result 5% of the time. That's what p-value means. This is also true regardless of sample size. What varies with sample size is how small the effect size has to be to be found significant; but, with large samples of pure noise, only small differences are likely; with small samples, bigger differences occur more often. Think of flipping a coin 10 times: Getting 8, 9 or even 10 heads wouldn't be absurd. However, if you toss a coin 1000 times, it would be truly odd to get 800 heads, much less 900 or 1000 (the exact numbers can be calculated, but that is not the point. However, with 1000 tosses, even a small deviation from 500 will be significant.

e.g. a t-test with random data, 2 vectors of length 10

set.seed(102811)
samp.size <- 10
t10 <- vector("numeric", 100)
for (i in 1:100){
x <- rnorm(samp.size)
y <- rnorm(samp.size)
t <- t.test(x,y)
t10[i] <- t$p.value
sum(t10 < .05)/100

I got 0.07

With two vectors of size 1000

set.seed(10291)
samp.size <- 1000
t1000 <- vector("numeric", 100)
for (i in 1:100){
  x <- rnorm(samp.size)
  y <- rnorm(samp.size)
  t <- t.test(x,y)
  t1000[i] <- t$p.value
}  
sum(t1000 < 0.05)/100

I got 0.05.

$\endgroup$
  • 1
    $\begingroup$ Florn, I find this well drafted, are there any academic papers / stats textbooks that one can reference that make a similar point? $\endgroup$ – SAFEX Jun 12 at 8:49
  • 1
    $\begingroup$ Which specific point? About looking at effect sizes? Or about what random is? $\endgroup$ – Peter Flom Jun 12 at 9:09
  • $\begingroup$ "What varies with sample size is how small the effect size has to be to be found significant", this is very intuitive from the text, but is there academic work that proves this point $\endgroup$ – SAFEX Jun 12 at 9:12
  • 1
    $\begingroup$ I don't know of a book that explicitly proves it - if you want some sort of mathematical statistics book, I am not the person to ask. I'm sure someone here will know, but they might not see this comment thread. Perhaps ask a separate question like "Explicit proof that what varies ...." $\endgroup$ – Peter Flom Jun 12 at 9:17
  • 1
    $\begingroup$ done and thanks again for the intuitive description (stats.stackexchange.com/questions/412643/…) $\endgroup$ – SAFEX Jun 12 at 9:23
2
$\begingroup$

As was already mentioned, in hypothesis testing you are actually investigating the null hypothesis, usually in the hope that you can reject it. In addition to the other answers I would like to propose a somewhat different approach.

Generally speaking, if you have some sort of a theory about what might be going on in your data, you could do a confirmatory analysis (like confirmatory factor analysis as just one example). In order to do so you would need a model. You can then see how well your model fits the data. This approach would also allow for testing different models against each other. The nice thing with Big Data is that it allows you to actually do these model tests. In contrast, in psychology e.g. it is often not really possible to do so, because the sample sizes tend to be too small for these kinds of methods.

I realize that typically with Big Data, an exploratory approach is used, because there is no theory, yet. Also, since I don't know what exactly you are interested in, this might not really be an option.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.