Are large data sets inappropriate for hypothesis testing? In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect.".
Well, I for one didn't know that. Is this true? Does it mean that hypothesis testing is worthless for large data sets?
 A: In a certain sense, [all] many null hypothesis are [always] false (the group of people living in houses with odd numbers does never exactly earn the same on average as the group of people living in houses with even numbers). 
In the frequentist framework, the question that is asked is whether the difference in income between the two group is larger than $T_{\alpha}n^{-0.5}$ (where $T_{\alpha}$ is the $\alpha$ quantile of the distribution of the test statistic under the null). Obviously, for $n$ growing without bounds, this band becomes increasingly easy to break through.
This is not a defect of statistical tests. Simply a consequence of the fact that without further information (a prior) we have that a large number of small inconsistencies with the null have to be taken as evidence against the null. No matter how trivial these inconsistencies turn out to be.
In large studies, it becomes then interesting to re-frame the issue as a bayesian test, i.e. ask oneself (for instance), what is $\hat{P}(|\bar{\mu}_1-\bar{\mu}_2|^2>\eta|\eta, X)$. 
A: Hypothesis testing for large data should the desired level of difference into account, rather than whether there is a difference or not. You're not interested in the H0 that the estimate is exactly 0. A general approach would be to test whether the difference between the null hypothesis and the observed value is larger than a given cut-off value. 
An simple example with the T-test:
You can make following assumptions for big sample sizes, given you have equal sample sizes and standard deviations in both groups, and $\bar{X_1} > \bar{X_2}$ :
$$T=\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}+\frac{\delta}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$
hence
$$T=\frac{\bar{X1}-\bar{X2}}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$
as your null hypothesis $H_0:\bar{X1}-\bar{X2} = \delta $ implies:
$$\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}\approx N(0,1)$$
This you can easily use to test for a significant and relevant difference. In R you can make use of the noncentrality parameter of the T distributions to generalize this result for smaller sample sizes as well. You should take into account that this is a one-sided test, the alternative $H_A$ is $\bar{X1}-\bar{X2} > \delta $.
mod.test <- function(x1,x2,dif,...){
    avg.x1 <- mean(x1)
    avg.x2 <- mean(x2)
    sd.x1 <- sd(x1)
    sd.x2 <- sd(x2)

    sd.comb <- sqrt((sd.x1^2+sd.x2^2)/2)
    n <- length(x1)
    t.val <- (abs(avg.x1-avg.x2))*sqrt(n)/sd.comb
    ncp <- (dif*sqrt(n)/sd.comb)
    p.val <- pt(t.val,n-1,ncp=ncp,lower.tail=FALSE)
    return(p.val)
}

n <- 5000

test1 <- replicate(100,
  t.test(rnorm(n),rnorm(n,0.05))$p.value)
table(test1<0.05)
test2 <- replicate(100,
  t.test(rnorm(n),rnorm(n,0.5))$p.value)
table(test2<0.05)

test3 <- replicate(100,
   mod.test(rnorm(n),rnorm(n,0.05),dif=0.3))
table(test3<0.05)

test4 <- replicate(100,
   mod.test(rnorm(n),rnorm(n,0.5),dif=0.3))
table(test4<0.05)

Which gives :
> table(test1<0.05)
FALSE  TRUE 
   24    76 

> table(test2<0.05)
TRUE 
 100 

> table(test3<0.05)
FALSE 
  100 

> table(test4<0.05)
TRUE 
 100 

A: The short answer is "no". Research on hypothesis testing in the asymptotic regime of infinite observations and multiple hypotheses has been very, very active in the past 15-20 years, because of microarray data and financial data applications. The long answer is in the course page of Stat 329, "Large-Scale Simultaneous Inference", taught in 2010 by Brad Efron. A full chapter (#2) is devoted to large-scale hypothesis testing.
A: "Does it mean that hypothesis testing is worthless for large data sets?"
No, it doesn't mean that. The general message is that decisions made after conducting a hypothesis test should always take into account the estimated effect size, and not only the p-value. Particularly, in experiments with very large sample sizes, this necessity to consider the effect size becomes dramatic. Of course, in general, users don't like this because the procedure becomes less "automatic".
Consider this simulation example. Suppose you have a random sample of 1 million observations from a standard normal distribution,
n <- 10^6
x <- rnorm(n)

and another random sample of 1 million observations from a normal distribution with mean equal to $0.01$ and variance equal to one.
y <- rnorm(n, mean = 0.01)

Comparing the means of the two populations with a t-test at the canonical $95\%$ confidence level, we get a tiny p-value of approximately $2.5\times 10^{-14}$.
t.test(x, y)

        Welch Two Sample t-test

data:  x and y
t = -7.6218, df = 1999984, p-value = 2.503e-14
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.013554059 -0.008009031
sample estimates:
   mean of x    mean of y 
0.0008947038 0.0116762485

It's correct to say that the t-test "detected" that the means of the two populations are different. But take a look at the very short $95\%$ confidence interval for the difference between the two population means: $[-0.013, -0.008]$.
Is a difference between the two population means of this order of magnitude relevant to the particular problem we are studying or not? 
A: I agree with the answers that have appeared, but would like to add that perhaps the question could be redirected.  Whether to test a hypothesis or not is a research question that ought, at least in general, be independent of how much data one has.  If you really need to test a hypothesis, do so, and don't be afraid of your ability to detect small effects.  But first ask whether that's part of your research objectives.
Now for some quibbles:


*

*Some null hypotheses are absolutely true by construction.  When you're testing a pseudorandom number generator for equidistribution, for instance, and that PRG is truly equidistributed (which would be a mathematical theorem), then the null holds.  Probably most of you can think of more interesting real-world examples arising from randomization in experiments where the treatment really does have no effect.  (I would hold out the entire literature on esp as an example. ;-)

*In a situation where a "simple" null is tested against a "compound" alternative, as in classic t-tests or z-tests, it typically takes a sample size proportional to $1/\epsilon^2$ to detect an effect size of $\epsilon$.  There's a practical upper bound to this in any study, implying there's a practical lower bound on a detectable effect size.  So, as a theoretical matter der Laan and Rose are correct, but we should take care in applying their conclusion.
A: I think its a problem of most significance tests having some general undefined class of implicit alternatives to the null, which we never know.  Often these classes may contain some sort of "sure thing" hypothesis, in which the data fits perfectly (i.e. a hypothesis of the form $H_{ST}:d_{1}=1.23,d_{2}=1.11,\dots$ where $d_{i}$ is the ith data point).  The value of the log likelihood is such an example of a significance test which has this property.
But one is usually not interested in these sure thing hypothesis.  If you think about what you actually want to do with the hypothesis test, you will soon recognise that you should only reject the null hypothesis if you have something better to replace it with.  Even if your null does not explain the data, there is no use in throwing it out, unless you have a replacement.  Now would you always replace the null with the "sure thing" hypothesis?  Probably not, because you can't use these "sure thing" hypothesis to generalise beyond your data set.  It's not much more than printing out your data.
So, what you should do is specify the hypothesis that you would actually be interested in acting on if they were true.  Then do the appropriate test for comparing those alternatives to each other - and not to some irrelevant class of hypothesis which you know to be false or unusable.
Take the simple case of testing the normal mean.  Now the true difference may be small, but adopting a position similar to that in @keith's answer, we simply test the mean at various discrete values that are of interest to us.  So for example, we could have $H_{0}:\mu=0$ vs $H_{1}:\mu\in\{\pm 1,\pm 2,\pm 3,\pm 4,\pm 5,\pm 6\}$.  The problem then transfers to looking at what level do we want to do these tests at.  This has a relation to the idea of effect size: at what level of graininess would have an influence on your decision making?  This may call for steps of size $0.5$ or $100$ or something else, depending on the meaning of the test and of the parameters.  For instance if you were comparing the average wealth of two groups, would anyone care if there was a difference of two dollars, even if it was 10,000 standard errors away from zero?  I know I wouldn't.
The conclusion is basically that you need to specify your hypothesis space - those hypothesis that you are actually interested in.  It seems that with big data, this becomes a very important thing to do, simply because your data has so much resolving power.  It also seems like it is important to compare like hypothesis - point with point, compound with compound - to get well behaved results.
A: No. It is true, that all useful point hypothesis tests are consistent and thus will show up a significant result if only the sample size is large enough and some irrelevant effect exists. To overcome this drawback of statistical hypotheses testing (already mentioned by the answer of Gaetan Lion above), there are relevance tests. These are similar to equivalence tests but even less common. For a relevance test, the size of a minimum relevant effect is prespecified. A relevance test can base on a confidence interval for the effect: If the confidence interval and the relevance region are disjoint, you may reject the null. 
However, van der Laan and Rose assume in their statement, that even true null hypotheses are tested in studies. If a null hypothesis is true, the propability to reject is not larger than alpha, especially in the case of large samples and even misspecified 
I can only see that the sample distribution is systematically different from the population distribution, 
A: The article you mention does have a valid point, as far as standard frequentist tests are concerned. That is why testing for a given effect size is very important. To illustrate, here is an anova between 3 groups, where group B slightly different than group A and C. try this in r:
treat_diff=0.001 #size of treatment difference
ns=c(10, 100, 1000, 10000, 100000, 1000000) #values for sample size per group considered
reps=10 #number of test repetitions for each sample size considered
p_mat=data.frame(n=factor(), p=double()) #create empty dataframe for outputs
for (n in ns){ #for each sample size
  for (i in c(1:reps)){ #repeat anova test ‘reps’ time
    treatA=data.frame(treatment="A", val=rnorm(n)) 
    treatB=data.frame(treatment="B", val=rnorm(n)+treat_diff) #this is the group that has the means slightly different from the other groups
    treatC=data.frame(treatment="C", val=rnorm(n))
    all_treatment=rbind(treatA, treatB, treatC)
    treatment_aov=aov(val~treatment, data=all_treatment)
    aov_summary=summary(treatment_aov)
    p=aov_summary[[1]][["Pr(>F)"]][1]
    temp_df=data.frame(n=n, p=p)
    p_mat=rbind(p_mat, temp_df)
  }
}

library(ggplot2)
p <- ggplot(p_mat, aes(factor(n), p))
p + geom_boxplot()

As expected, with greater number of samples per test, the statistical significance of the test increases:

A: Hypothesis testing traditionally focused on p values to derive statistical significance when alpha is less than 0.05 has a major weakness.  And, that is that with a large enough sample size any experiment can eventually reject the null hypothesis and detect trivially small differences that turn out to be statistically significant.
This is the reason why drug companies structure clinical trials to obtain FDA approval with very large samples.  The large sample will reduce the standard error to close to zero.  This in turn will artificially boost the t stat and commensurately lower the p value to close to 0%. 
I gather within scientific communities that are not corrupted by economic incentives and related conflict of interest hypothesis testing is moving away from any p value measurements towards Effect Size measurements.  This is because the unit of statistical distance or differentiation in Effect Size analysis is the standard deviation instead of the standard error.  And, the standard deviation is completely independent from the sample size.  The standard error on the other hand is totally dependent from the sample size. 
So, anyone who is skeptical of hypothesis testing reaching statistically significant results based on large samples and p value related methodologies is right to be skeptical.  They should rerun the analysis using the same data but using instead Effect Size statistical tests.  And, then observe if the Effect Size is deemed material or not.  By doing so, you could observe that a bunch of differences that are statistically significant are associated with Effect Size that are immaterial.  That's what clinical trial researchers sometimes mean when a result is statistically significant but not "clinically significant."  They mean by that one treatment may be better than placebo, but the difference is so marginal that it would make no difference to the patient within a clinical context.    
A: I think what they mean is that one often makes an assumption about the probability density of the null hypothesis which has a 'simple' form but does not correspond to the true probability density. 
Now with small data sets, you might not have enough sensitivity to see this effect but with a large enough data set you will reject the null hypothesis and conclude that there is a new effect instead of concluding that your assumption about the null hypothesis is wrong.
A: A (frequentist) hypothesis test, precisely, address the question of the probability of the observed data or something more extreme would be likely assuming the null hypothesis is true.  This interpretation is indifferent to sample size.  That interpretation is valid whether the sample is of size 5 or 1,000,000.
An important caveat is that the test is only relevant to sampling errors.  Any errors of measurement, sampling problems,coverage, data entry errors, etc are outside of the scope of sampling error.  As sample size increases, non-sampling errors become more influential as small departures can produce significant departures from the random sampling model.  As a result, tests of significance become less useful.
This is in no way an indictment of significance testing.  However, we need to be careful about our attributions.  A result may be statistically significant.  However, we need to be cautious about how we make attributions when sample size is large.  Is that difference due to our hypothesized generating process vis a vis sampling error or is it the result of any of a number of possible non-sampling errors that could influence the test statistic (which the statistic does not account for)?
Another consideration with large samples is the practical significance of a result.  A significant test might suggest (even if we can rule out non-sampling error) a difference that is trivial in a practical sense.  Even if that result is unlikely given the sampling model, is it significant in the context of the problem?  Given a large enough sample, a difference in a few dollars might be enough to produce a result that is statistically significant when comparing income among two groups.  Is this important in any meaningful sense?  Statistical significance is no replacement for good judgment and subject matter knowledge.
As an aside, the null is neither true nor false.  It is a model.  It is an assumption.  We assume the null is true and assess our sample in terms of that assumption.  If our sample would be unlikely given this assumption, we place more trust in our alternative.  To question whether or not a null is ever true in practice is a misunderstanding of the logic of significance testing.
A: One simple point not made directly in another answer is that it's simply not true that "all null hypotheses are false."
The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.
But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than $\alpha$ (the tests's bound on false positives).
The medical industry tests "non-inferiority" hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the $\alpha$, usually 0.05).
A: It is not true.  If the null hypothesis is true then it will not be rejected more frequently at large sample sizes than small. There is an erroneous rejection rate that's usually set to 0.05 (alpha) but it is independent of sample size. Therefore, taken literally the statement is false. Nevertheless, it's possible that in some situations (even whole fields) all nulls are false and therefore all will be rejected if N is high enough. But is this a bad thing?
What is true is that trivially small effects can be found to be "significant" with very large sample sizes.  That does not suggest that you shouldn't have such large samples sizes.  What it means is that the way you interpret your finding is dependent upon the effect size and sensitivity of the test.  If you have a very small effect size and highly sensitive test you have to recognize that the statistically significant finding may not be meaningful or useful.
Given some people don't believe that a test of the null hypothesis, when the null is true, always has an error rate equal to the cutoff point selected for any sample size, here's a simple simulation in R proving the point. Make N as large as you like and the rate of Type I errors will remain constant.
# number of subjects in each condition
n <- 100
# number of replications of the study in order to check the Type I error rate
nsamp <- 10000

ps <- replicate(nsamp, {
    #population mean = 0, sd = 1 for both samples, therefore, no real effect
    y1 <- rnorm(n, 0, 1) 
    y2 <- rnorm(n, 0, 1)
    tt <- t.test(y1, y2, var.equal = TRUE)
    tt$p.value
})
sum(ps < .05) / nsamp

# ~ .05 no matter how big n is. Note particularly that it is not an increasing value always finding effects when n is very large.

A: Isn't all this a matter of type I error versus type II error (or power) ? If one fixes the type I error probability ($\alpha$) at 0.05, then , obviously (except in the discrete case), it will be 0.05 whether the sample is large or not.  
But for a given type I error probability, 0.05 e.g., the power, or the probability that you will detect the effect when it is there (so the probability to reject $H_0$ (=detect the effect) when $H_1$ is true (=when the effect is there)), is larger for large sample sizes. 
Power increases with sample size (all other things equal). 
But the statement that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect." is incorrect. 
A: "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true—will declare a statistically significant effect". 
Well, in a sense all (most) Null hypothesis are false. The parameter under consideration has to be equal to the hypothesized value right down to an infinite number of decimal points which is an absolute rarity. So it is highly likely that the test will declare a statistically significant effect as sample size increases.
A: This is a critic of Bayesian inference, a different way to thing statistics (different from frequentists form that everyone learns in others courses others than statistics, we learn both).
"The critic is that you can proof anything with a large sample because gives to you a p-value."
That's why we look in A LOT of others metrics, aic,f,rmse,anova..... 
None of my professors answered me about how to lead this thing just "make a sample, so your dataset will be small and this will not happen" 
But I'm not happy with that, but I use this way :/ 
