Best statistical model/technique for assessing part data

Question

I'm a software engineer who has been tasked with doing statistical analysis on data collected from a physical test system. The system collects data on a part by setting input conditions, reading in measured data, then changing the input conditions and doing another measurement. A typical data collection file ramps through a couple of dimensions of input data collecting output data for each.

We already have a comparison routine that compares two files to one another. I now need to add the capablity to add two new compare capabilities to the program. Both are batch compare capabilities that would compare a bunch of files. In one case, all files would be compared to a control file with data collected on a known good part. We want to generate some kind of quality measure that would show how close the data sets are to the control data set.

In the other mode, we assume that most of the data files are on good parts, but some are not. We need to do a compare each file to one another and then norm the result into a bell curve.

I did a lot of math in school and even tutored it for 4 years in college. However statistics was never part of my formal education. For some reason my engineering program didn't think engineers needed to know statistics. I have picked up some bits and pieces here and there, but I'm not sure what type of statistical analysis is best for this task.

The data set for a control file (known good part) looks something like this:

Input   Measured Out
.1  .05
.2  .06
.3  .07
etc.

A part that would test good might look something like this

Input   Measured Out
.1  .051
.2  .059
.3  .072
etc.

But a bad part might look like

Input   Measured Out
.1  .08
.2  .09
.3  .10
etc.

I need some measure that would indicate the good part compared to the control has some sort of measure of high quality, and the bad part is of low quality. Most of what I've found applies to psychological data and most of what I've learned about statistics is related to psychology (I have a bit of interest in the subject and my SO is a Psychologist who barely passed statistics in college). The nature of this sort of data is somewhat different from a psychological data set though. The data have dependencies that psychological data don't have for one thing.

Can someone point me to the best statistical technique for this problem?

Answer 1

It looks like the files consist of discrete values that are compared. So what you want is a measure of closeness. One common statistical measure is mean square error (the average of the squared deviation for the number in the test file to the one in the control file. You could then pick a threshold C such that you would call the part good if the MSE is less than C and bad if it is greater.

Answer 2

For first mode , we can use Paired comparisons.

This can also be done by simulation. Here the concept is that suppose for input 0.1 we have value of 0.5 in one file and 0.52 in other file. So we will shuffle these two outputs. Similarly we will do it for all outputs in both files. Then we will find the difference in the means of both files. And we will simulate this expermient 10000 times , and we will get a bell curve of difference of means. Now we will check how many times we will get the difference greater than the actual difference of means in both files. If this value is greater than some threshold value 0.05(i.e 5% or any value thought by you, according to your experiment), then we will say that the new file is not as good as our control file. You can set threshold value according to your need, how much you can afford for your product to vary from good to bad.

For second mode you can make a bell curve and can emit those files which are very much varying from most probable files..