I wanted to ask this question on stackoverflow but I think it is more suitable here. If I am wrong, please tell me.
My question concerns the use in statistics to analyse physical/engineering data. As statistical tools to be used depend on the context and are rarely universal, it is mandatory that I describe my problem which is rather long.
My question
In my case, how should I valuate the residual to consider how good are my tests? Should I simply compare their residual mean squares (RMS)? Or should I compare the square root of the RMSs? Or should I compare the root square of the sum of the square of the residual devided by the residual degrees of freedom? Or anything else?
summary: $$ \frac{\sum_i(Y_i-\widehat{Y})^2}{n-2} or \sqrt{\frac{\sum_i(Y_i-\widehat{Y})^2}{n-2}} or \frac{\sqrt{\sum_i(Y_i-\widehat{Y})^2}}{n-2} or \; ELSE \;? $$ With the context you will understand why I don't want to use statistical tools using the mean $\overline{Y}$ and only residuals.
Settings
My real problem is rather complex and concerns analysis of some physics-related test data. So for the sake of the example and to avoid to go into physics (which is not our topic here): let's say I am a quality engineer and I want to check the quality of a set of M samples by tensile test. (I picked tensile test because it is taught early in High school and rather simple test)
My test is simple: I pull each of my sample only and strictly in the elastic regime which has normally a linear correlation between stress (load L, Newtons) and strain to the m (displacement D, Meters). m is the power factor of the displacement and is unknown. So I have the following law (1):
$L=a*D^m $ (1)
$1\leq m \leq 2$
note: for m=1, a is the spring stiffness "k", related to its Young's modulus "E", and I have a typical tensile test with a typical sample. It is not important if it is the case here but it could happen. m should be about the same for all my samples but is never strictly the same (you know, experiments...) and is unknown.
My instrument saves the displacement and load data points in a $2*n$ matrix, n being the number of data points. I have a total of M samples, hence M tests.
Objective
My goal is to check if each test follow correctly my law (1). If yes, my sample is good. If not, my sample is bad and I can trash it (in the recycling bin of course). When plotted, tests on "bad" samples would show steps or several regimes when $L$ vs $D^m$ for $\forall m$ (it is clearly visible for both cases). "bad" samples simply mean there are structural defaults. For good samples, $L$ vs $D^m$ (with best $m$) is a simple line (minus errors induced by the measuring instrument).
Algorithm
To find m, I use a simple iteration process with a linear least square method with Matlab. For $L=0$, $D=0$ of course (no force, no strain). Here is the simple algorithm (not in perfect matlab language for clarity!) for one test:
[RMS,m,a] = function(D,L) %inputs: Displacement D and load L
%outputs: RMS, m and a
Vm=1*101 Vector %m will have a 0.01 precision.
Vr=1*101 Vector
Va=1*101 Vector
for 1<Vm<2
X_m = D.^Vm
Va=X_m\L %left matrix divide built-in function of matlab:
%Solution by least square method used.
Vr=sum((L-a*X_m).^2)./(length(L)-2) %Residual Mean Square calculation
%Note: length(L)=n: number of data points
end for
[RMS, RMSPos]=min(Vr) %RMS = Residual Mean Square = min(r):least residual.
%RMSPos: position in the vector of the least residual
m=Vm(RMSPos) %Saves the m with the least residual
a=Va(RMSPos) %Saves the a with the least residual
end function
And I do that for each of my M tests.
Issue
Now I want to define a method to say "these samples are good" or "these samples are bad". I would compare the $M$ RMS I got from the $M$ tests: the ones close to 0 would tell me about the good tests, the ones bigger than 0 would tell me about the bad ones. I could decide of a RMS limit $RMS_{lim}$ where $RMS\leq RMS_{lim}$ would give me the good samples and $RMS > RMS_{lim}$ would give me the bad samples. In some ways, I can also sort the "goodness" of all of my $M$ samples by sorting from the smallest residuals to the biggest ones. This way, I don't need to check the plot of all of my $M$ tests manually and see if I have a beautiful line or an ugly random thingy.
While I can definitely compare the RMSs within one test for different '$m$'s, I am wondering if it is both physically and mathematically acceptable to compare the RMSs between the different tests. Why? Because on the one hand my different test solutions have similar but NOT strictly equal max $L$ (or $L_n$), max $D$ (or $D_n$), '$m$'s, '$a$'s and '$n$'s ($n$: number of data points in one test). In top of that RMS is a squared value, so I should probably compare the root square of the RMS between the tests. Particularly if it is below 1...
What to compare then?
Side-notes
I cannot use the tools using the mean, like for example the coefficient of determination, because it would take into account the distance between each data point and the mean of all of them. The issue? Imagine I decide to rerun one test and "forget" one $(D_i,L_i)$ couple close to the previously calculated mean and pull a bit more my sample, still in the elastic regime, to get a higher maximum Load $L_n$. I keep the same amount of data (same $n$). What happens? Well the RMS will still be about the same as my sample will still be intact (elastic test, remember? :) ). But the coefficient of determination would get closer to one... Which has no sense as my sample didn't become suddenly "better". I tried it experimentally to make it sure.
Then: in no way this is a method really used for filtering good devices from bad devices. Don't worry, no airplane will crash or building will self-destroy because a bad engineer (me) used a bad method to test the quality of its components :) . I am just a naive student with too little mathematical/statistical background working on a -rather- complex topic.
Acknowledgement
Thank you! To whomever could read this long text.
Wli
