I am interested to find a way to identify if two sets of data can be considered statistically different at 95% Confidence level (or any other). To be more specific, my data sets are composed of 5 values. They correspond to 5 readings of the same detector by one microscope. So, two detectors with 5 readings each can represent the same value or not. The problem is to find a method to answer this question. I'm wondering if a t-test is the proper tool. Am I right? I would like to do the statistical test in R.

  • 1
    $\begingroup$ Before going to sattistics, I think you should first answer why these values may be different. Are they completelly wrong values, or they represent close values that differ due to measurement errors ?! Are these 5 readings related to each other somehow ?! $\endgroup$ – BeS Feb 23 '15 at 15:31

I am slightly unsure what you are testing for. Are you testing for concordance (you expect the two sets of data to be the same) or are you testing to find a difference between the two sets?

For Difference Between Groups If you have only a single, 2-level categorical predictor, and hypothesize there is a difference in means for the outcome, then a t-test is appropriate. However, I would strongly recommend against doing more than 1 t-test. The Type I error rate of hypothesis testing goes up rapidly with many t-tests. In R:

eggs <- rnorm(5)
spam <- rnorm(5)


The confidence interval (-2.6, 1.4) includes zero, so the null hypothesis cannot be rejected and no difference is observed.

EDIT: As @gung mentioned below, with only 5 reads, the assumptions of the independent samples t-test cannot really be checked. t-test assumes both samples are normally and identically distributed. For your data, this might be unreasonable. If so, Mann-Whitney's U test is more appropriate. It is a nonparametric analog to the t-test. In R:


Here, the p-value is ~1.0, so a similar conclusion is made that the null cannot be rejected and no difference is observed.

For Concordance However, if you are hypothesizing the more subtle outcome that the two sets of reads should be equivalent (being the same reads from the detectors on the same microscope), I would recommend using a concordance measure. Lin's Concordance measures how well two sets of paired data concord with each other. The statistic of interest, rho, is similar to Pearson's product-moment correlation coefficient, but adjusted for exact agreement along the x=y line (note, this is R, not R-square). Closer to 1.0 is strong concordance; closer to 0.0 is no concordance. In R, use the epiR package.


Here, the rho is modest, 0.38, indicating low concordance. The confidence interval (-0.30, 0.80) includes 0, so the null hypothesis (the reads are discordant) cannot be rejected.

These two results seem to contradict each other, but it depends on the subtlety of what you're asking and what sort of assumption is appropriate. In the first, we assumed they are the same; in the second, we assumed they are different. Depends on what question you are trying to answer and the type of assumptions that are reasonable for your study.

  • 2
    $\begingroup$ Welcome to the site, @Joshua. This is a good answer; I hope we'll see more. I have one issue, however, w/ so few readings a lot will depend on assumptions that cannot be checked. It may be safer to use the Mann-Whitney U-test. What do you think? $\endgroup$ – gung - Reinstate Monica Feb 23 '15 at 18:31
  • 2
    $\begingroup$ Thank you, @gung! I've read/learned a lot here, and I do hope to start contributing. You bring up a great point that the assumptions cannot be checked, so it falls back on the OP's confidence in the t-test assumptions for his/her data. Mann-Whitney is certainly safer, and I will edit above to include that. $\endgroup$ – Ashe Feb 23 '15 at 18:59
  • $\begingroup$ Thanks for your very valuable answer Joshua. I've been looking for response to my question before reading your answer and I got similar conclusion as you. I use the book "Probability and statistics with R (Maria Dolores Ugarte, Ana F. Militino, and Alan Arnholt)". I am a physicist with very background on statistics. Thank you very much $\endgroup$ – user3329055 Feb 25 '15 at 14:30
  • $\begingroup$ I'm glad it was useful @user3329055! If you felt it answered your needs well enough to be considered accepted, please feel free to click that as well :) Thanks! $\endgroup$ – Ashe Feb 25 '15 at 15:37

@Joshua and @PaulGowder both provided links and detailed answers but unfortunately both answers are wrong in the sense that they do not answer the OP question as stated in the title. In the title, the OP wants to provide evidence that the two set of measures are EQUAL not different. t-test or Mann-Whitney will provide evidence (with 95% confidence) that the two sets are NOT EQUAL - but it cannot provide evidence that they are equal. Confusingly, in the first line of the post, the OP asks how to verify that the sets are different (for which the answers posted are correct).

Showing that two sets are not different with 95% of confidence is NOT the same as showing that they are they are equal (with 95%). To "prove" that two sets are EQUAL you need an equivalence-test tag: in CV. In particular TOST (two one sided t-test) is the "most common" test, but I do not know what to do with the fact that you only have 5 data points.

Let me link to one of the great answers in CV on equivalence testing: How to test hypothesis of no group differences? You should start from there. There are others. Sometimes this topic is also described as "proving the null hypothesis"

  • 3
    $\begingroup$ This is a reasonable point, but it is awfully strongly stated. I think the question itself is ambiguous; the title suggests 1 interpretation but the body suggests another. @Joshua addressed the ambiguity well & his suggestion of agreement is a reasonable 1 (although I am also aware of TOST). $\endgroup$ – gung - Reinstate Monica Feb 23 '15 at 19:48
  • $\begingroup$ While the answer here - equivalance testing - is definitely one that should be raised, the complaint that the other answers are literally wrong may be overstating the case. Frequently the phrasing of questions are made without careful consideration of the subtle difference (in ordinary English at least) between testing for equality and testing for a difference. It's true that sometimes when people make a post like that they are seeking something like equivalence testing, but in many cases that's actually not what's being sought. The other answers may well respond to the underlying question... $\endgroup$ – Glen_b -Reinstate Monica Feb 23 '15 at 22:46
  • $\begingroup$ (ctd)... Indeed the other answerers might as easily point to "if two sets of data can be considered statistically different" in the first sentence and say that this answer is "WRONG" on that basis. I think that the word - especially in capitalized form - is much too strong as the question stands, and this answer would be better without it. $\endgroup$ – Glen_b -Reinstate Monica Feb 23 '15 at 22:50
  • $\begingroup$ I downgraded the force of my statements, but I think the gist of the criticism should remain. A user that arrives at the question because of the title will receive an inappropriate answer (and will make a serious mistake). The query "How to find out if two sets of data are equal" on Google gives this question as the first entry!! $\endgroup$ – Jacques Wainer Feb 24 '15 at 2:11
  • $\begingroup$ Good point, especially after reducing the strength of critique (+1). However, if you want to fully answer the question as stated (the OP asks for R solution), it would be better, if you would provide some references to R packages/functions and, even better, an MRE code in R. $\endgroup$ – Aleksandr Blekh Feb 24 '15 at 2:27

Here's a helpful discussion: http://www.researchgate.net/post/What_is_the_difference_between_T-test_F-Test_and_anova_tests_in_statistics

If you're just comparing means across two groups, e.g., to test an experimental effect, the basic default choice is a t-test.

  • 2
    $\begingroup$ Welcome to the site, @PaulGowder. We prefer if you can give a summary of the useful information at a link here, in case the link goes dead & so that potential readers can decide if it's something they want to pursue. $\endgroup$ – gung - Reinstate Monica Feb 23 '15 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.