Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

If I have one clock, I know what the time is.

If I have two clocks, I now am unsure.

Although simplified, the problem is related to a more complex system I am testing.

If I have the following measurements from two systems:

Attribute   System 1   System 2
A           0          0
B           10         100
C           90         100
D           100        110
E           900        1000

then I'd like a way to determine some value of error between the systems (assuming I cannot distinguish which system, if any, is correct)

So

  • attribute A values are considered no-error or both systems correct
  • attribute B values are considered a larger error than A, C, D and E values
  • attributes C, D and E values are have the same error (or similar error)

Also, assume the attributes are independent.

My understanding of statistics is low, and I can only recall some things about relative error, but iirc I require knowing whether one system is correct to determine the relative error of the other system.

How does one go about resolving this issue?

share|improve this question
I"m a little confused by the wording. You say, "determine some value of error between the systems". Do you mean "difference" between the two systems, or "error" of one of the systems? Or something different, like trying to figure out what "true" value is and thus how each differs from this true value? – Wayne Jan 21 '12 at 16:32
@Wayne the last thing you said, although knowing the true value is not so important (it helps though), just how much error each system is off by. – jamiroquai Jan 23 '12 at 1:35
Ah, pretty much impossible, as far as I can tell. I'll +1 rolando2's answer. At best, if you have many systems, you could average their values for each attribute and then say how far off each actual machine is from the average. – Wayne Jan 23 '12 at 14:15
One more question: are you wanting to compare N products at one time -- say for a purchase decision -- or products over time? In the latter case, there's a field called Statistical Process Control, which has charts that let you see if your process seems to be shifting over time (i.e. a part of your manufacturing process has broken). – Wayne Jan 23 '12 at 14:52

2 Answers

up vote 3 down vote accepted

Sorry to disappoint you, but I don't see any definitive statistical solution to your problem. There's no glorious trick that can determine where truth lies and which of your systems is closer to it. So what can you do? Obtain multiple measurements from each system and average them... Average the measurements you have and treat those averages as best estimates... Devise additional systems, making your problem potentially better and potentially worse.

share|improve this answer
I figured there wouldn't be some magical silver bullet. – jamiroquai Jan 23 '12 at 1:36
1  
@rolando2: agreed that if the OP has many systems, average them and then state how far each machine is from the average. I'd recommend also calculating the standard deviation for each attribute, so that readers can determine if System 32 is really out of line on Attribute A, or not. – Wayne Jan 23 '12 at 14:17
up vote 1 down vote

A few things that might help (a bit).

  1. You could calculate the coefficient of variation for each attribute. This would allow you to at least quantify what you are saying when you say (for example) "the error in Attribute B is higher than the error of the other attributes".

  2. You could use analysis of variance to quantify how much of the variation is between attributes, and how much is between your two measuring systems.

  3. You could model the system2 measurements on the system1 measurements and hence get a quantification of the relationship between the two (crudely speaking, it would come up with something like "system 2 on average returns 10% higher estimates than system 1").

None of these completely fix your problem but they give you some different angles to look at it.

share|improve this answer
Good ideas for a tough situation. – rolando2 Jan 22 '12 at 0:27
Good options (I like having options). I'd have to look into these in a little more detail and see if they help. – jamiroquai Jan 23 '12 at 1:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.