Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 7 down vote favorite

Please pardon me if this question is not clear. I am not sure if I am using the right terminologies.

I have conducted an experiment in different environments multiple times. So my data looks something like this:

Environment1  1.2  2.1  1.1  1.5  1.6
Environment2  4.2  2.6  3.5  2.5  2.9
Environment3  7.2  4.6  5.3  4.5  1.6
Environment4  0.0  0.0  1.2  15.0 0.0
Environment5  3.2  2.4  7.2  5.5  6.6
Environment6  23.2  32.1  18.1  1.5  19.6

I can clearly see (or maybe my intuition says) that the experiment was not conducted properly in Environment4 (too low and fluctuating a lot) and Environment5 (way too high) but am not sure how to prove this. Am I supposed to rely on hypothesis-testing with the hypothesis:

The experiment was not conducted properly in Environments 4 and 6.

and then use some procedure to prove this? Or is there a standard way of showing this? Can someone please help me how to approach this kind of problems? I am using R.

share|improve this question
Nice question, its a good example to expose to different procedures, because we basically know without any maths or formality, that Environment 4 and 6 are different to the rest (and Environment 1 is a little different from 2, 3, and 5). Thus any good procedure should be able to produce the obvious result, only difference coming from quantifying how different in a mathematical sense. The obvious question is "is there any other way the experiment could have actually produce these results, besides an error?" – probabilityislogic Jan 30 '11 at 6:28
@probabilityislogic: Thank You. What you say is useful: if I can somehow quantify the effectiveness of the experiment in each environment, then may be I can say something but am still not sure what to say or how to say. Ah.. (...feeling stupid typing in puzzles) :) Regarding your question: the experiment was quite controlled in the sense that, it was made sure that the environment did not change. However, the procedure could have gone wrong. May be the procedure was not executed properly according to the guidelines (perhaps?) – Legend Jan 30 '11 at 6:35
I'm talking more along the lines of "is $32.1$ a physically meaningful quantity? What would happen in the real world if this were correct." It may also be useful to speak to someone who actually did the experiment 4 or 6 (preferably the person who recorded the data). – probabilityislogic Jan 30 '11 at 14:31
@probabilityislogic: I see. I get your point. The data is question is a response time variable. My take on your question would be that the value does make sense in a physical world but its just too unusual enough to be called a rare case. The person I talked to said he did not do anything differently. Actually, the data that I put here is just a sample from the entire data and there are some cases like this spread here and there. – Legend Feb 1 '11 at 0:17
so it would appear that the most likely result is an error, but interesting discoveries can be made if you "dig deeper" so to speak. Could possibly be a new finding of some sort! but don't get too excited, it probably is nothing, but it may be worthwhile to entertain the possibility, and see where it leads you. – probabilityislogic Feb 1 '11 at 2:29

1 Answer

up vote 2 down vote accepted

You can do a student test to see if the mean is different between the group 4,6 and the rest. Even if your sample size is small you will conclude in a difference. Note that it will tell you that group 4,6 is significantly different in average from the rest but it won't tell you that "The experiment was not conducted properly in Environments 4 and 6" which can't be answered without a knowledge of what "properly" means in the observations.

share|improve this answer
girad: Actually, this question came up from someone over the testing team. Properly means that they were given a set of instructions to execute to get a final value. The experiment will complete even if one of the instructions is skipped but will result in an incorrect observation. I will check out the student test that you mentioned. But if the test relies on mean, isn't mean supposed to be a bad measure due to its sensitivity to change in the data values? Thank you for your time. – Legend Jan 30 '11 at 6:52
@Legend A test of difference of means may be inappropriate, but that is not the fault of @robin, as pointed out in the second half of his response, which is apt: the test to use is determined by which characteristic of a suite of results signals an "improper" experiment. You could conduct an F-test for a difference of standard deviations; you could conduct multiple-outlier tests; you could conduct a Kruskal-Wallis test; etc., depending on what kind of differences you're looking for. – whuber♦ Jan 30 '11 at 18:08
@Legend There is also another difficulty that is shadowed by your question because here you guessed that 4,6 was the different samples. But what if you don't know in advance... you will have to test all configuration and probably introduce a multiple hypothesis criterion. In this case this looks like outliers detection and many question have already dealt with that here. – robin girard Jan 31 '11 at 7:35
@whuber: I did not intend to see it is anyone's fault. I am a novice here so I apologize if sounded so. @robin girard: That is a very interesting take. Thanks. I was just thinking about outlier detection. Will you be able to point me to some relevant material for this particular case? All I have used before are simple ones like k-means etc. – Legend Feb 1 '11 at 0:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.