3
$\begingroup$

I have two predictions from two different types of methods. "predictedHousePrices1" is a continuous variable and the output of a prediction from a RandomForest model, "predictedHousePrices2" is the output of the predict() function a RidgeRegression model. I'd like to compare which one better explains the variability in the real data. I'm wondering if a likelihood ratio test is the best way to do this, for example:

m1 <- lm(realHousePrices~predictedHousePrices1) # R^2 = 0.25
m2 <- lm(realHousePrices~predictedHousePrices2) # R^2 = 0.30

Is it correct to use a likelihood ratio test to check if an $R^2$ of .25 is "significantly" more than 0.30, for example:

m3 <- lm(realHousePrices~predictedHousePrices1+predictedHousePrices2)
library("epicalc")
lrtest(m3, m1)

Or is there a better way to do this?

$\endgroup$
3
$\begingroup$

The likelihood-ratio test is appropriate only if the two models you are comparing are nested, i. e., if one can be retrieved from the other, e. g., by fixing parameters (e. g., to zero). Models with more parameters will always fit better, the question the LR test answers is whether the increase in fit is defensible given the amount of added parameters.

If you want to compare non-nested models, you may use information criteria such as AIC or BIC (the smaller the better).

Whether R² = .3 is better then .25 also depends largely on your field, substantive theory, and the variables in the models. If m1 has only one parameter but m2 has 20, the first may be preferable. If m1 misses a theoretically very relevant predictor, you may choose m2 although an increase of .05 may sound not that much.

Just to make sure: Your predictedValues is just a dummy for one or more predictors, i.e., variables, right? Because otherwise m3 does not make much sense, to me at least.

$\endgroup$
  • 3
    $\begingroup$ You can compare both models to a model containing both predictors, and obtain two likelihood ratio tests. More information about that is in my book Regression Modeling Strategies. $\endgroup$ – Frank Harrell Jul 23 '13 at 11:28
  • $\begingroup$ Hi Hajope, "predictedValues1 & 2" are continuous variables and the output of two completely different types of modelling method. One is predicted using RandomForests, the other using Ridge Regression. So for example, they could be predicted house prices. RealValues could be considered to be the actual house prices. My question is whether "predictedValues2" is doing a better job of explaining the variability in RealValues than is "predictedValues1". $\endgroup$ – user21262 Jul 23 '13 at 15:24
  • $\begingroup$ Just to make sure: If you want to use the LR test (how and when, see above), your model statements need to include the actual variables, e.g., lm(Y~X1+X2+X3) and not the predictedValues you get from m1$fitted.values. Because otherwise, the LR test can not correctly count the number of parameters, which you need for the dfs. $\endgroup$ – hplieninger Jul 24 '13 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.