# How to compare performance of multiple regression models?

I am taking a little data analysis course using SAS software and I need help with pretty much the basics.

There is this full description in the American Statistical Association page.

Basically I have the data obtained from caught fish (species, weight, length etc)

    Obs    Spec  Wt        Lt1     Lt2     Lt3     HtP    WhP    Sex
----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+-
1      1     242.0     23.2    25.4    30.0    38.4   13.4   NA
2      1     290.0     24.0    26.3    31.2    40.0   13.8   NA
3      1     340.0     23.9    26.5    31.1    39.8   15.1   NA
4      1     363.0     26.3    29.0    33.5    38.0   13.3   NA
5      1     430.0     26.5    29.0    34.0    36.6   15.1   NA
...


I want to predict weight using other known characteristics. How can I prove of disprove that the same regression function would work for all fish species? Would checking hypothesis that the weight distributions (for different species) are statistically different be enough?

Then I have a bunch of models apparently suggested by experts

Weight=a+b*(Length3*Height*Width)+epsilon
Log(Weight)=a+b1*Length3+epsilon
Weight^(1/3)=a+b1*Length3+epsilon
Log(Weight)=a+b1*Length3+b2*Height+b3*Width+epsilon
Weight^(1/3)=a+b1*Length3+b2*Height+b3*Width+epsilon
Weight=a*Length3^b1*Height^b2*Width^b3+epsilon


How should I compare them? Simply look for the best root mean square error?

## 1 Answer

Ans 1: You can check to see how well your models fit your data. Some will fit well and some will not. It does not mean that you disprove a model, this is meaningless. Some models will outperform the others.

Ans 2: In general, it is not enough. You need to double check other assumptions of the model. Also see the Ans 3 below.

Ans 3,4: There are different ways to compare these models. One of them would be based on their mean square errors (MSE). This means that you are looking at "in sample performance" of the models. In addition to MSE, you can check the "out of sample performance" of these models. It means that you remove some observations from your given data, and re-fit all these models to this new smaller data set. Then by using the obtained models, you try to predict the weights for the knows values of the characteristics for each model. Since you know the observed value of the weights, you can figure out how these models will actually perform.

In addition, I can see that these models have different number of parameters. In general, the model with more parameters, is expected to fit better than the model with less number of parameter. But there should be a balance between the number of parameters and the goodness of fit. So one way is to use some criteria like, Akaike information criterion (AIC) or Bayesian information criterion (BIC). In these criteria, there is a penalty for each new parameter that is added to the model. In general, the models with less AIC or BIC would be the preferred model.