1
$\begingroup$

Let's say I have some machine that measures some real-world phenomenon and outputs some raw data. The raw data need some (elementary) processing before it can be converted to a quantitative output, say $Y_1$.

It is thought that $Y_1$ is a linear response to some specific real-world quantity, say $X_1$, so that:

$Y_1 = a_1 \cdot X_1 + b_1 \tag 1$

As a side note, I have exact values for $X_1$, which can be measured directly, but the direct method is normally impractical.

I would like to propose that the raw data measured by the machine is more meaningful to process in a different way to produce a new readout, say $Y_2$, so that:

$Y_2 = a_2 \cdot X_1 + b_2 \tag 2$

Furthermore, I want to show, that when another variable, representing a measure of noise, is added, say $X_2$, the model is enhanced:

$Y_2 = a_2 \cdot X_1 + X_2 + b_2 \tag 3$

Ideally I want to test the superiority of model 3 against model 1 - is this possible?

If not, can I at least test 2 against 1?

|cite|improve this question
$\endgroup$
  • $\begingroup$ By the way I asked this question, although expressed differently, at the StataList couple of days ago, but so far no responses $\endgroup$ – Rozh Oct 23 '15 at 21:28
  • $\begingroup$ Are Y1 & Y2 measured in the same units (eg, cm)? Are they measures of the same construct? Do you have data on X1 or not? What are these variables? Can you be more concrete? I'm not sure this question is answerable at present. $\endgroup$ – gung - Reinstate Monica♦ Oct 23 '15 at 22:02
  • $\begingroup$ gung: 1) Y1 and Y2 are not measured on the same units, and the coefficient of variation (SD/mean) of the two are also quite different. 2) They are both acquired using manipulation of the same raw data, but they are quite different both in terms of value and physical interpretation. 3) Yes, as I wrote above, I have "exact" values for all X1, which were measured using a separate method. 4) all variables are continuos and unrestricted. 4) I have described the problem in general terms to avoid the use of field-specific technical terms. I appreciate your help $\endgroup$ – Rozh Oct 24 '15 at 9:36
  • $\begingroup$ Link to discussion at the StataList, as mentioned above: statalist.org/forums/forum/general-stata-discussion/general/… $\endgroup$ – Rozh Oct 24 '15 at 14:43
1
$\begingroup$

My understanding is that we cannot adequately compare models that aren't nested - at least with standard statistical tests.

However, if you're merely seeing if Model 1 or Model 2 is more appropriate, I might calculate the $R^2$ and see which model explains more variability in $Y$.

I don't believe you can test Model 1 against Model 3 directly, but since Model 2 is nested in Model 3, I might do the standard F-test for nested models. This will test $H_0$: Model 2 versus the alternative $H_A$: Model 3. If Model 3 is not seen as superior to Model 2, then you could use Model 2 and test that against Model 1. However since your results of this entire process are either inconclusive (i.e. Model 3 is superior to Model 2 and cannot be compared to Model 1) or conclusive but only for cases where Model 1 or Model 2 would "win," there are likely statistical issues of which I'm not currently thinking.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks Matt. The R2 is indeed higher in model 3 compared to 2, which again is higher than model 1. But how do I test if the difference in R2 is real (significant) or just by chance ? $\endgroup$ – Rozh Oct 24 '15 at 14:45
  • $\begingroup$ Well, you want to do an F-test between Model 2 and Model 3, like I said above. That will tell you whether Model 3 is significantly better than Model 2. $\endgroup$ – Matt Brems Oct 24 '15 at 14:46
  • $\begingroup$ Once you do that, you can compare the model you select from your F-test with Model 1. I don't think you can do a statistically significant test, but this may be the best you can do with different outcome variables. $\endgroup$ – Matt Brems Oct 24 '15 at 14:47
  • $\begingroup$ OK, so if I understand correctly, the winner of 3 vs 2 cannot be tested (as in statistical test) vs 1. So the comparison of the winner of 2vs3 and model 1 must be done through e.g. logical arguments. Is this correct ? $\endgroup$ – Rozh Oct 24 '15 at 14:59
  • $\begingroup$ To my understanding, yes. I'm only familiar with using statistical tests for nested models. I haven't had much experience with testing models with different dependent variables. I believe this is more of a "Which $Y$ is more appropriate?" question, which may be best examined using metrics like $R^2$ and looking at scatterplots to assess linear relationships between the dependent and independent variables. $\endgroup$ – Matt Brems Oct 24 '15 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.