I have two sets of data that hypothesis tests have shown to be normal and from the same distribution. I'm using MATLAB and for the way they give p-values, higher p-values suggest a better goodness-of-fit, as I understand it. I want to combine the two data sets to get the parameters for the combined data set. However, when I run a hypothesis test on the combined data set, hypothesis tests either reject the null assumption of Normality or come very close as seen by the test statistics. I've been looking for an explanation but have no idea where to start. Could anyone shed some light into this?

  • $\begingroup$ Hypothesis tests don't show to distributions are the same. Presumably you mean to say something else. Are you talking about failing to reject some hypothesis there? Can you show QQ plots for the two sets of data? $\endgroup$ – Glen_b -Reinstate Monica Jan 21 '14 at 9:43

If you're talking about a test of the null hypothesis that your sample data come from an exactly normal distribution, you'll probably want to see this question: Is normality testing 'essentially useless'?

Most (if not all) real data is not normally distributed exactly. You can estimate the skewness and kurtosis to quantify how different your data's distribution is from a normal distribution. Even small amounts of skew or excess kurtosis become easier to estimate precisely as data aggregates. By combining your two samples, you're improving the precision of your estimates. It seems you have enough data to estimate your distribution's differences from a normal distribution so precisely that, with your normality tests' $p$ values, you can say there's "very close" to or less than $\alpha$ (typically 5%) chance that, from a truly normal distribution, you would randomly sample a dataset that is as large and at least as different from a normal distribution as yours is. Since that's the null hypothesis in a nutshell, and it seems very unlikely to be true, you're probably right to reject it, but this is somewhat of a foregone conclusion.

Whether your data are too different from a normal distribution for your intended purposes is another question. The hypothesis tests you're using probably can't answer it for you by themselves: their outcomes will depend on your sample size, but won't take into account the sensitivity of whatever other analyses you have planned, among other reasons. You would probably get a better idea of how important this issue is by studying the sensitivity of your planned analyses to violations of the normality assumption; some really aren't very sensitive to the usual kinds of minor violations that are present in approximately normal data. You might get better sense of how non-normal your data are from a Q-Q plot, as @Glen_b suggested; knowing just how much non-normality is too much can still be difficult to tell from these plots though.

If you have time, or if your data are particularly problematic, you might want to look into robust alternatives to any other analyses you plan to run. This might even be one of the shorter routes to avoiding validity problems while ignoring the idiosyncrasies of your distribution, but bear in mind that those idiosyncrasies are sometimes interesting in their own right! Finally, if you're particularly serious about minding the sensitivity of any subsequent analyses to data that aren't from exactly normal distributions, you might want to consider doing some simulation testing of those analyses too.

(Full disclosure: for this answer, I borrowed and modified some text from a previous answer of mine.)

Edit: The null hypothesis of the two-sample Kolmogorov-Smirnov test states that the two samples are drawn from populations with the exact same distribution (or are both drawn from the same population, which is roughly equivalent). This would mean equal distributional parameters too...but to test normality (as you say you have), one of the distributions (your "reference distribution") would have to be a normal distribution. You'd need to run two such tests to compare each of two distributions to a normal distribution, or to each other and then one of them to a normal distribution. Testing twice inflates your overall false rejection rate, so you would want to adjust for multiple tests before rejecting those nulls (which you haven't anyway, so this is somewhat moot). Wikipedia mentions a multivariate version of this test, but it's relatively new, so I doubt you would've performed this version of the test unknowingly.

  • $\begingroup$ Thanks so much. I think I see what you are saying. So what does one do in this situation? Would one use a linear estimator to get the mean and variance of the combined data set? $\endgroup$ – TSP Jan 21 '14 at 10:22
  • $\begingroup$ It depends on what you're after. The mean and variance of the combined set will be just that. If you're trying to estimate a population mean and variance for the underlying distribution, I think your sample statistics would still be the best point estimates...but if you've got other plans in mind for your estimates of the population's central tendency and variability, it might not be so simple. Are you only after the mean and variance parameters? For their own sake? $\endgroup$ – Nick Stauner Jan 21 '14 at 10:37
  • 1
    $\begingroup$ I'm comparing samples drawn from two days and each data point can be considered independent within each day and between days. My thinking was to first see if the two samples were drawn from the same distribution and the two-sample Kolmogorov-Smirnov test seemed to indicate that they were. Then I tested to see if both samples could be considered drawn from a Normal distribution. Testing both samples seemed to support this. So my thinking was, if I combined the two datasets, I could get more precise estimates. Is this even valid? $\endgroup$ – TSP Jan 21 '14 at 11:02
  • $\begingroup$ Again, it depends on what you're really trying to estimate. If you're trying to estimate the distributional parameters of a population that you are defining as inclusive of these samples from two separate days, then combining those samples should give you more valid estimates of that population's parameters. If the day the data were collected or other circumstances that might've changed across those two days might matter for the way you're defining your population of interest (e.g., if something particularly strange happened on either day), that could threaten validity; otherwise, seems fine. $\endgroup$ – Nick Stauner Jan 21 '14 at 11:13

Your interpretation of the $p$-value suggests a fundamental misunderstanding about the nature of your hypothesis test. A test cannot ever confirm that the null is true. At best, it can only state that there is insufficient evidence to suggest that the alternative hypothesis is true.

If you get a "high" $p$-value using a goodness-of-fit test or a test of normality, that doesn't mean the evidence is strong that the data are drawn from the same distribution, because the $p$-value is a conditional probability: it is the probability of observing a result as extreme as that obtained, given $H_0$ is true. So, at best, all your test has been able to show is that the data you observed does not suggest any reason to believe they are drawn from different distributions--this is not the same as saying they are in fact drawn from the same distribution.

This subtlety becomes all the more apparent when we consider that the nature of the tests you are using here tend to have very low power: that is, even if the samples are drawn from different distributions, you would need a LOT of observations for the test to affirmatively conclude that they are.

  • $\begingroup$ Thanks. It seems to be that way from what I've been reading about the Kolmogorov-Smirnov test, but then is there a more sensitive or powerful test available? $\endgroup$ – TSP Jan 21 '14 at 12:43
  • 1
    $\begingroup$ Here's the other sticking point. You are essentially testing two different things. The first KS test is seeing if the samples are drawn from the same distribution, regardless if that distribution is actually normal. The second KS test on the pooled sample is testing if the pooled sample is normal. Obviously, you can fail to reject on the first test even if the second test rejects--e.g., you can draw from IID uniform distributions. That is of course unless you also ran individual KS tests of normality on each sample. $\endgroup$ – heropup Jan 21 '14 at 12:56
  • 1
    $\begingroup$ Regarding using the KS test for testing normality, the Shapiro-Wilk test has better power compared to KS used for this purpose. However, Shapiro-Wilk does not test for distributional similarity; an alternative is the Cramer-von Mises test. Although I don't know if it has better power than the two-sample KS test, I would not be surprised if it does: KS has very low power for small sample sizes. $\endgroup$ – heropup Jan 21 '14 at 12:59

Combining two normally distributed dataset into one, can result in a not-normally distributed dataset. This is best illustrated by an example:

(supposing you're working with R)

# creat two normal distributed datasets
x <- data.frame(var=rnorm(5000,2,1)) # 5000 observations, mean = 2, sd = 1
y <- data.frame(var=rnorm(5000,6,1)) # 5000 observations, mean = 6, sd = 1

When you make density plots the look like this for x and y respectively:

X density plot

Y density plot

Both look rather normally distrubed. Now combine the two datasets.

# combining the datasets
z <- rbind(x,y)

When you make a density plot of z, you can see that it doesn't look like a normal distribution at all:

Z density plot

  • 1
    $\begingroup$ Sorry, I should have been more clear. I meant to say that I ran a two-sample Kolmogorov-Smirnov test to see if the two samples could be considered drawn from the same distribution. Wouldn't that mean that the distribution and its parameters would be the same? I may not be entirely understanding the test. $\endgroup$ – TSP Jan 21 '14 at 10:15
  • $\begingroup$ Why did you choose the Kolmogorov-Smirnov test? $\endgroup$ – Jaap Jan 21 '14 at 10:39
  • $\begingroup$ It was the only one I knew of that could compare the distributions of two-samples without making assumptions about the underlying distributions of said samples. My thinking was that if the two samples were both proven by goodness-of-fit tests to be Normal and both were proven to be drawn from the same distribution, then combining the two would give parameters having higher precision. $\endgroup$ – TSP Jan 21 '14 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.