Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have two large data sets, in fact, one of them is even much larger than the other.

Visually, there doesn't seem to be that much difference between them:

comparing gene targets from different algorithms depending upon whether they have a miRNA inside

The actual data underlying the box plot isn't normally distributed and doesn't normalise well to transformations. They are roughly the same distribution (i.e. the YES and NO distributions for each algorithm), but the large data size differences make other tests a bit useless. I have applied the Two-sample Kolmogorov-Smirnov test, however this is probably wrong and it gives extremely significant results.

My questions are:

1) Do statistical tests on large datasets produce significant results given even slight differences between the two samples? The 'slightness' being magnified given huge data points.

2) Is visual inspection better with large datasets rather than applying non-parametric and parametric tests in which certain underlying assumptions may be violated.

3) For this data, what is the best course of action?

Edit

My data has structure like :

My data is of the form:

Name    Bind    miRNA
a       300     NO
b       500     YES
c       140     YES
d       2345    NO
share|improve this question
    
is this a bivariate dataset? If so, why not just use a scatterplot? Also, you have bivariate boxplots 'bagplot', why not use them? –  user603 Apr 7 at 12:19
    
The data isn't bivariate, but I'll have a look at bagplot now. Thanks. –  Mark Ramotowski Apr 8 at 10:14
    
but if the data is not biariate, how dod you manage to plot it on a boxplot? Did you leave relevant features out? –  user603 Apr 9 at 12:27
    
Maybe I don't understand the term bivariate, I assumed it means 2 variables. I have the name variable, the bind variable and the miRNA bool. –  Mark Ramotowski Apr 9 at 17:01
    
No problem, my question was actually wrongly asked. Yes, your data is bi (=two) variate. But only one of those variables (Bind) is continuous, so you can't use a bagplot (weal, you can, but then it will just revert to an ordinary boxplot). –  user603 Apr 10 at 8:19

3 Answers 3

up vote 11 down vote accepted

I suggest summarizing the difference with a general robust measure that does not depend on normality: the concordance probability that comes from the Wilcoxon-Mann-Whitney two-sample test. The concordance proportion estimates the probability that a randomly chosen value from group A exceeds a randomly chosen value from group B. This can be generalized to your "pairing of pairs" setting in which you can estimate the probability that method 1 provides measurements that are "more concordant" between A and B than method 2 does. This is implemented in the R Hmisc package rcorrp.cens function.

share|improve this answer
  1. Yes. This is one key problem with standard goodness-of-fit tests on large datasets.
  2. I would prefer visual inspection, as well as measures of effect size. Even if there is a large overlap in distributions, a 15% improvement in some KPI may be very useful. I wouldn't care too much about specific distributions, depending on your specific application. In addition, boxplots are rather crude ways of displaying data. Here are a few alternatives.

  3. Hard to say, since we don't know your data... My suggestion about effect sizes may already be helpful.

share|improve this answer

There is nothing wrong with statistical summaries of large data sets. If a method would be appropriate with N = 100 then it is appropriate with N = 100,000 or 100,000,000.

There is, however, something wrong with how most people interpret p-values. The answer to your first question is "yes", but that answer is just another indication that you should look at effect size, not p value.

Regarding your second question: Visual inspection is critical with large or small data sets. But numeric comparison is very useful with both, as well. Their utility is not a function of data set size.

Regarding your third question: Use both visual and numeric comparisons. Choose both the visual and numeric method that 1) are suited to your data and 2) are suited to the questions you want to ask about your data.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.