2
$\begingroup$

I'm a software engineer who has been tasked with doing statistical analysis on data collected from a physical test system. The system collects data on a part by setting input conditions, reading in measured data, then changing the input conditions and doing another measurement. A typical data collection file ramps through a couple of dimensions of input data collecting output data for each.

We already have a comparison routine that compares two files to one another. I now need to add the capablity to add two new compare capabilities to the program. Both are batch compare capabilities that would compare a bunch of files. In one case, all files would be compared to a control file with data collected on a known good part. We want to generate some kind of quality measure that would show how close the data sets are to the control data set.

In the other mode, we assume that most of the data files are on good parts, but some are not. We need to do a compare each file to one another and then norm the result into a bell curve.

I did a lot of math in school and even tutored it for 4 years in college. However statistics was never part of my formal education. For some reason my engineering program didn't think engineers needed to know statistics. I have picked up some bits and pieces here and there, but I'm not sure what type of statistical analysis is best for this task.

The data set for a control file (known good part) looks something like this:

Input   Measured Out
.1  .05
.2  .06
.3  .07
etc.

A part that would test good might look something like this

Input   Measured Out
.1  .051
.2  .059
.3  .072
etc.

But a bad part might look like

Input   Measured Out
.1  .08
.2  .09
.3  .10
etc.

I need some measure that would indicate the good part compared to the control has some sort of measure of high quality, and the bad part is of low quality. Most of what I've found applies to psychological data and most of what I've learned about statistics is related to psychology (I have a bit of interest in the subject and my SO is a Psychologist who barely passed statistics in college). The nature of this sort of data is somewhat different from a psychological data set though. The data have dependencies that psychological data don't have for one thing.

Can someone point me to the best statistical technique for this problem?

$\endgroup$

2 Answers 2

3
$\begingroup$

It looks like the files consist of discrete values that are compared. So what you want is a measure of closeness. One common statistical measure is mean square error (the average of the squared deviation for the number in the test file to the one in the control file. You could then pick a threshold C such that you would call the part good if the MSE is less than C and bad if it is greater.

$\endgroup$
1
  • $\begingroup$ I assume that for the case where we are comparing to a known good file (data set), the known good would be declared the mean from the start. For the case where we don't have a known good file, then I would get the mean of each data point across all files and then compare each file to that mean file? We will probably need to experiment a bit to figure out what the best threshold values would be. But that is a matter of lab experiments once we get the code in place. Thanks a bunch for your help, Bill $\endgroup$
    – Bill Olson
    Aug 17, 2012 at 2:51
2
$\begingroup$

For first mode , we can use Paired comparisons.

This can also be done by simulation. Here the concept is that suppose for input 0.1 we have value of 0.5 in one file and 0.52 in other file. So we will shuffle these two outputs. Similarly we will do it for all outputs in both files. Then we will find the difference in the means of both files. And we will simulate this expermient 10000 times , and we will get a bell curve of difference of means. Now we will check how many times we will get the difference greater than the actual difference of means in both files. If this value is greater than some threshold value 0.05(i.e 5% or any value thought by you, according to your experiment), then we will say that the new file is not as good as our control file. You can set threshold value according to your need, how much you can afford for your product to vary from good to bad.

For second mode you can make a bell curve and can emit those files which are very much varying from most probable files..

$\endgroup$
2
  • $\begingroup$ I'm a bit unclear what you mean by shuffle the outputs? By means of both files, do you mean getting the mean data point for the whole file? Adding up all data points and getting the average? Maybe it's my weak understanding of the terminology, but I'm a bit lost with your explanation. My customer would welcome more than one comparison mode. So if I come up with more than one statistical method, they would be happy. $\endgroup$
    – Bill Olson
    Aug 17, 2012 at 3:34
  • $\begingroup$ The method which i have written is just a simulated way of doing paired t-test. So you can use paired t-test , in place of this. $\endgroup$ Aug 25, 2012 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.