2
$\begingroup$

If I have one clock, I know what the time is.

If I have two clocks, I now am unsure.

Although simplified, the problem is related to a more complex system I am testing.

If I have the following measurements from two systems:

Attribute   System 1   System 2
A           0          0
B           10         100
C           90         100
D           100        110
E           900        1000

then I'd like a way to determine some value of error between the systems (assuming I cannot distinguish which system, if any, is correct)

So

  • attribute A values are considered no-error or both systems correct
  • attribute B values are considered a larger error than A, C, D and E values
  • attributes C, D and E values are have the same error (or similar error)

Also, assume the attributes are independent.

My understanding of statistics is low, and I can only recall some things about relative error, but iirc I require knowing whether one system is correct to determine the relative error of the other system.

How does one go about resolving this issue?

Improve this question
$\endgroup$
4
  • $\begingroup$ I"m a little confused by the wording. You say, "determine some value of error between the systems". Do you mean "difference" between the two systems, or "error" of one of the systems? Or something different, like trying to figure out what "true" value is and thus how each differs from this true value? $\endgroup$
    – Wayne
    Commented Jan 21, 2012 at 16:32
  • $\begingroup$ @Wayne the last thing you said, although knowing the true value is not so important (it helps though), just how much error each system is off by. $\endgroup$
    – jamiroquai
    Commented Jan 23, 2012 at 1:35
  • $\begingroup$ Ah, pretty much impossible, as far as I can tell. I'll +1 rolando2's answer. At best, if you have many systems, you could average their values for each attribute and then say how far off each actual machine is from the average. $\endgroup$
    – Wayne
    Commented Jan 23, 2012 at 14:15
  • $\begingroup$ One more question: are you wanting to compare N products at one time -- say for a purchase decision -- or products over time? In the latter case, there's a field called Statistical Process Control, which has charts that let you see if your process seems to be shifting over time (i.e. a part of your manufacturing process has broken). $\endgroup$
    – Wayne
    Commented Jan 23, 2012 at 14:52

2 Answers 2

Reset to default
4
$\begingroup$

Sorry to disappoint you, but I don't see any definitive statistical solution to your problem. There's no glorious trick that can determine where truth lies and which of your systems is closer to it. So what can you do? Obtain multiple measurements from each system and average them... Average the measurements you have and treat those averages as best estimates... Devise additional systems, making your problem potentially better and potentially worse.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ @rolando2: agreed that if the OP has many systems, average them and then state how far each machine is from the average. I'd recommend also calculating the standard deviation for each attribute, so that readers can determine if System 32 is really out of line on Attribute A, or not. $\endgroup$
    – Wayne
    Commented Jan 23, 2012 at 14:17
1
$\begingroup$

A few things that might help (a bit).

  1. You could calculate the coefficient of variation for each attribute. This would allow you to at least quantify what you are saying when you say (for example) "the error in Attribute B is higher than the error of the other attributes".

  2. You could use analysis of variance to quantify how much of the variation is between attributes, and how much is between your two measuring systems.

  3. You could model the system2 measurements on the system1 measurements and hence get a quantification of the relationship between the two (crudely speaking, it would come up with something like "system 2 on average returns 10% higher estimates than system 1").

None of these completely fix your problem but they give you some different angles to look at it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Good ideas for a tough situation. $\endgroup$
    – rolando2
    Commented Jan 22, 2012 at 0:27
  • $\begingroup$ Good options (I like having options). I'd have to look into these in a little more detail and see if they help. $\endgroup$
    – jamiroquai
    Commented Jan 23, 2012 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.