Hypothesis Testing with Big Data

Question

How do you perform hypothesis tests with big data? I wrote the following MATLAB script to emphasize my confusion. All it does is generate two random series, and run a simple linear regression of one variable on the other. It performs this regression several times using different random values and reports averages. What tends to happen is as I increase the sample size, p-values on average get very small.

I know that because the power of a test increases with sample size, given a large enough sample, p-values will become small enough, even with random data, to reject any hypothesis test. I asked around and some people said that with 'Big Data' its more important to look at the effect size, ie. whether the test is significant AND has an effect large enough for us to care about. This is because in large sample sizes the p-values will pick up of very small differences, like it is explained here.

However, the effect size can determined by scaling of the data. Below I scale the explanatory variable to a small enough magnitude that given a large enough sample size, it has large significant effect on the dependent variable.

So I'm wondering, how do we gain any insight from Big Data if these problems exist?

%make average
%decide from how many values to make average
obs_inside_average = 100;

%make average counter
average_count = 1;

for average_i = 1:obs_inside_average,






%do regression loop
%number of observations
n = 1000;

%first independent variable (constant term)
x(1:10,1) = 1; 

%create dependent variable and the one regressor
for i = 1:10,

    y(i,1) = 100 + 100*rand();

    x(i,2) = 0.1*rand();

end





%calculate coefficients
beta = (x'*x)\x'*y;

%calculate residuals
u = y - x*beta;

%calcuatate sum of squares residuals
s_2 = (n-2)\u'*u;

%calculate t-statistics
design = s_2*inv(x'*x);

%calculate standard errors
stn_err = [sqrt(design(1,1));sqrt(design(2,2))];

%calculate t-statistics
t_stat(1,1) = sqrt(design(1,1))\(beta(1,1) - 0);
t_stat(2,1) = sqrt(design(2,2))\(beta(2,1) - 0);

%calculate p-statistics
p_val(1,1) = 2*(1 - tcdf(abs(t_stat(1,1)), n-2));
p_val(2,1) = 2*(1 - tcdf(abs(t_stat(2,1)), n-2));






%save first beta to data column 1
data(average_i,1) = beta(1,1);

%save second beta to data column 2
data(average_i,2) = beta(2,1);

%save first s.e. to data column 3
data(average_i,3) = stn_err(1,1);

%save second s.e. to data column 4
data(average_i,4) = stn_err(2,1);

%save first t-stat to data column 5
data(average_i,5) = t_stat(1,1);

%save second t-stat to data column 6
data(average_i,6) = t_stat(2,1);

%save first p-val to data column 7
data(average_i,7) = p_val(1,1);

%save second p-val to data column 8
data(average_i,8) = p_val(2,1);

end

%calculate first and second beta average
b1_average = mean(data(:,1));
b2_average = mean(data(:,2));

beta = [b1_average;b2_average];

%calculate first and second s.e. average
se1_average = mean(data(:,3));
se2_average = mean(data(:,4));

stn_err = [se1_average;se2_average];

%calculate first and second t-stat average
t1_average = mean(data(:,5));
t2_average = mean(data(:,6));

t_stat = [t1_average;t2_average];

%calculate first and second p-val average
p1_average = mean(data(:,7));
p2_average = mean(data(:,8));

p_val = [p1_average;p2_average];

beta
stn_err
t_stat
p_val

Answer 1

As Peter suggested, I think one of the important things in the era of "Big Data" is to put even less emphasis on p-values, and more on an estimation of the magnitude of effect.

Some of my own work struggles with this in ways I think are even more insidious than with Big Data - for stochastic computational models, your power is entirely a function of patience and computing resources. It's an artificial construct.

So turn back to the effect estimate. Even if its significant, does a 0.0001% increase in something matter in the real world?

I've also been playing around with reversing some of the ideas behind reporting study power. Instead of reporting the power your study had to detect the observed effect, reporting the minimum effect size the study was powered to find. That way the reader can know if significance was essentially guaranteed.

Answer 2

The insight that you desire is going to come from confidence intervals, not as much from p-values. With a very large sample size you are going to get a very precise confidence intervals, provided your statistical assumptions are correct.

Answer 3

It is important to look at effect size regardless of whether the data are big or small.

With purely random data, you should get a significant result 5% of the time. That's what p-value means. This is also true regardless of sample size. What varies with sample size is how small the effect size has to be to be found significant; but, with large samples of pure noise, only small differences are likely; with small samples, bigger differences occur more often. Think of flipping a coin 10 times: Getting 8, 9 or even 10 heads wouldn't be absurd. However, if you toss a coin 1000 times, it would be truly odd to get 800 heads, much less 900 or 1000 (the exact numbers can be calculated, but that is not the point. However, with 1000 tosses, even a small deviation from 500 will be significant.

e.g. a t-test with random data, 2 vectors of length 10

set.seed(102811)
samp.size <- 10
t10 <- vector("numeric", 100)
for (i in 1:100){
x <- rnorm(samp.size)
y <- rnorm(samp.size)
t <- t.test(x,y)
t10[i] <- t$p.value
sum(t10 < .05)/100

I got 0.07

With two vectors of size 1000

set.seed(10291)
samp.size <- 1000
t1000 <- vector("numeric", 100)
for (i in 1:100){
  x <- rnorm(samp.size)
  y <- rnorm(samp.size)
  t <- t.test(x,y)
  t1000[i] <- t$p.value
}  
sum(t1000 < 0.05)/100

I got 0.05.

Answer 4

As was already mentioned, in hypothesis testing you are actually investigating the null hypothesis, usually in the hope that you can reject it. In addition to the other answers I would like to propose a somewhat different approach.

Generally speaking, if you have some sort of a theory about what might be going on in your data, you could do a confirmatory analysis (like confirmatory factor analysis as just one example). In order to do so you would need a model. You can then see how well your model fits the data. This approach would also allow for testing different models against each other. The nice thing with Big Data is that it allows you to actually do these model tests. In contrast, in psychology e.g. it is often not really possible to do so, because the sample sizes tend to be too small for these kinds of methods.

I realize that typically with Big Data, an exploratory approach is used, because there is no theory, yet. Also, since I don't know what exactly you are interested in, this might not really be an option.