0
$\begingroup$

I have two models M1 and M2 that I use to predict Utility. Here's how they look with Utility on the x-axis.

M1: enter image description here

M2: enter image description here

Now despite some evidence of clustering, M1 clearly must be the superior product since it moves well with Utility across its range of values while M2 predicts the same minimum or near-minimum value for over a third of all points, leading to very little variation at low-to-medium values of Utility. When running linear best fits, M1 still has the superior R-squared, but only barely so because M2's is significant just by virtue of fitting a near-horizontal line to capture this non-variability and taking advantage of wildly asymmetrical residuals.

My question is what metric can be used to meaningfully compare the two to demonstrate the usefulness of M1 and/or penalize M2 for its asymmetry and basically assigning a uniform predicted value.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First, the dependent variable is usually put on the Y axis.

A linear fit with model M2 is completely inappropriate, so comparing R^2 between the two is not a good idea. The relationship with M1 also appears non-linear, although it's not as obvious.

I'd fit a more flexible model (like restricted cubic splines) with both variables and then compare those.

You can also compare the residuals for the two models.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am having a hard time detecting any hint of nonlinearity in M1. The conditional mean of $X$, given $Y$, looks like a linear function of $Y$. The conditional distributions even look reasonably homoscedastic. Perhaps the unconventional switching of the axes along with the clustering of data at low values of $Y$ are creating your impression of nonlinearity. $\endgroup$
    – whuber ♦
    Commented Feb 3, 2016 at 21:38
  • $\begingroup$ To me, it looks like there is a curve. The slope looks lower at the lower levels and then appears to be about a 45 degree line. The odd axes make it odd, for sure. A loess line would help. $\endgroup$
    – Peter Flom
    Commented Feb 4, 2016 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.