I want to compare the fit of a linear model (M1) and nonlinear model (M2):
- M1: $y = b_0 + b_1x_1 + b_2x_2 + b_3x_1x_2 + \epsilon, \epsilon \sim N(0, \sigma^2)$
- M2: $y = b_0 + b_1x_1 + b_2x_2 + b_1 b_2x_1x_2 + \epsilon, \epsilon \sim N(0, \sigma^2)$
In particular I want to know whether M1 is significantly different from M2.
To estimate the parameters, I am minimizing the least-squares errors rather than maximizing the likelihood through MLE procedures. In particular, I am using the R function nls() as follow:
# Creating a sample data set n <- 50 x1 <- rnorm(n, 1:n, 0.5) x2 <- rnorm(n, 1:n, 0.5) b0 <- 1 b1 <- 0.5 b2 <- 0.2 y <- b0 + b1*x1 + b2*x2 + b1*b2*x1*x2 + rnorm(n,0,0.1) # Actual model fit M1 <- nls(y ~ b0 + b1*x1 + b2*x2 + b3*x1*x2, start=list(b0=1, b1=0.5, b2=0.5, b3=0.5)) M2 <- nls(y ~ b0 + b1*x1 + b2*x2 + b1*b2*x1*x2, start=list(b0=1, b1=0.5, b2=0.5))
I want to compare the models using a measure of relative fit such as AIC, which can be done in R as follow:
AIC(M1, M2) df AIC M1 5 -88.47849 M2 4 -90.46491
Because $\Delta AIC \approx 2$ and the models differ by only one parameter, I would conclude that both of them fit the data similarly well.
In addition, I want to know whether the parameter $b_3$ from M1 significantly add to the fit using a statistical test such as an F-test. This can be done in R as follow:
anova(M1, M2) Analysis of Variance Table Model 1: y ~ b0 + b1 * x1 + b2 * x2 + b3 * x1 * x2 Model 2: y ~ b0 + b1 * x1 + b2 * x2 + b1 * b2 * x1 * x2 Res.Df Res.Sum Sq Df Sum Sq F value Pr(>F) 1 46 0.40843 2 47 0.40855 -1 -0.00011097 0.0125 0.9115
My general question is:
- Are these analyses appropriate?
- Can I use AIC to compare least-squares fitted models?
From a few posts such as this one it looks like AIC should be appropriate. However, I've seen posts such as this one that indicates that using AIC on non-MLE fitted models might be a problem. I understand that least-squares is equivalent to MLE if the error is normally distributed, but is this true even for non-linear models?
- Can I use a F-test to test whether $b_3$ is significantly different from $b_1 b_2$?
I know such F-test makes sense if the model are nested, but I'm unsure whether it is appropriate in this case.