0
$\begingroup$

I have two variables (A and B). A has 3,000 samples; B has 500,000. They represent cumulative solar radiation (summer 2000) in Alta Valtellina (Italy), divided in two major areas (A and B) which I need to compare. Below are the histograms of my datasets: histogram of samples A (left) and B (right)

As far as I know, a two-sample t-test won't be ideal here because my samples are too large (see my previous question). So I thought about using the Cohen's D test. Running the kstest I know the distribution of these variables is not normal (output is 1, meaning "rejecting the Null Hypothesis that data is normally distributed"). Does the Cohen's D test needs the data to be normally distributed (I could not find a reference for this)?. If so, should I have to transform my data, or would someone suggest a better test for my case?

EDIT

At present I am writing a scientific paper, and only with a good justification like a statistical test (or a statistical comparison) I woul be able (or not) to say that my datasets differ from each other at a certain degree. If they are different (what I expect from the figure above, but I might be wrong), I would use the statistics of A (which is a sample of solar radiation pixels of a large area) to classifiy the remaining pixels of my study area (if they are similar to A I would assign them the same classification of A; B otherwise).

$\endgroup$
7
  • 1
    $\begingroup$ If you think your samples are "too large" for a hypothesis test, you probably shouldn't be using hypothesis tests even if the sample were smaller. The samples being large doesn't indicate a problem with testing as such, it simply highlights more clearly that it answers a different question than the one you really intend to ask. $\endgroup$ – Glen_b Sep 9 '15 at 4:45
  • $\begingroup$ As I want to find "large" differences in my datasets rather than looking at "tiny" ones, I think I'd better use the Cohen's D rather than the t-test, as the latter works with standard errors which gets really small for large dataset (denominator is high -> SE gets tiny). Would the Cohen's D be a reasonable choice in my case? Also, please take a look at my previous question (linked above), to better understand why I might prefer not using the t-test for my case (the situation is similar), but I am afraid I could not use the Cohen's D if my data is not normally distributed, could I? $\endgroup$ – umbe1987 Sep 9 '15 at 7:30
  • 1
    $\begingroup$ Note that $\bar{x}_1$ and $\bar{x}_2$ will be so near to normally distributed as makes no odds. Further note that Cohen's d is actually $\frac{\bar{x_1}-\bar{x_2}}{\sigma}$ and you don't know $\sigma$ (i.e. Cohen's $d$ as he defined it - see Cohen (1992), Table 1** - is not a sample quantity!). Even if you ignore what Cohen is actually doing and replace $\sigma$ by $s$, in what sense does "Cohen's d" qualify being called a test rather than simply an estimate of the effect size? Does Cohen's arbitrary classification into small, medium and large really correspond to what you need here?...ctd $\endgroup$ – Glen_b Sep 9 '15 at 8:19
  • 1
    $\begingroup$ ctd... ** see also Cohen "Statistical power analysis for the behavioural sciences", 2e., p274, where he explicitly states that $\sigma$ in Cohen's $d$ is a population quantity. What Cohen is doing is discussing effect size in the context of a power analysis, which is conducted before the data are even collected. What you're doing is calculating something after the data are collected. $\endgroup$ – Glen_b Sep 9 '15 at 8:36
  • 1
    $\begingroup$ The point I was making about Cohen is that you're not using a test, you're simply making a comparison. Are you after an actual hypothesis test, or not? $\endgroup$ – Glen_b Sep 9 '15 at 22:34
1
$\begingroup$

The sample size being too large for a T-test? That's unheard of. You've misunderstood what the linked post is getting at.

To put it plainly, the T-test works very nicely for testing the hypothesis that the means of the respective populations are different when the sample size is large. The only point of further consideration is simply a matter of interpretation: are the differences you find practically different?

Indeed, large large samples are adequately powered to detect tiny, tiny differences: differences that are essentially useless when it comes to what you're measuring, like blood pressure lowering of 0.1 nmHg (not usually enough to combat hypertensions), weight loss of 0.05 kg (small compared to day-to-day water weight), hairgrowth of 1nm/day, etc etc.

For that reason, you should look at the 95% confidence interval and discuss the range of possible effects and what they imply.

As an anecdote, in my favorite TV show Mythbusters, they tested the hypothesis that (American) footballs filled with helium fly farther when thrown. They did a test and found a consistent difference favoring helium for longer flight. However, the difference was only a couple of inches compared to the dozens of yards that a remotely competant QB is capable of lobbing the ball. Therefore they concluded that there really wasn't a difference, since it's not practically significant.

$\endgroup$
1
  • $\begingroup$ Thanks for posting. I am not interested in tiny differences, rather I am looking for "significant" difference in my dataset. If A and B range 0-500 W m^-2 in value, I want to know if A differs (e.g.) 50 W m^-2 from B, or viceversa. This is just an example, but from the linked post I got that using the t-test is ok to search for tiny difference, while the Cohen's D is more suitable to look for "larger" difference as it works with standard deviations rather than standard errors (as the t-test). $\endgroup$ – umbe1987 Sep 9 '15 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.