How to check if my regression model is good

Question

One way to find accuracy of the logistic regression model using 'glm' is to find AUC plot. How to check the same for regression model found with continuous response variable (family = 'gaussian')?

What methods are used to check how well does my regression model fit the data?

Answer 1

I would suggest a brief search on "linear regression model diagnostics" as a start. But here are some that I would suggest you to check:

Make sure the assumptions are satisfactorily met

• Use scatterplot or component plus residual plot to examine the linear relationship between the independent predictor(s) and the dependent variable.

• Compose a plot with standardized residual versus predicted value and ensure there isn't extreme point with very high residual, and the spread of the residual is largely similar along the predicted value, as well as spreading largely equally above and below the mean of residual, zero.

• You can also change the y-axis to residual$^2$. This plot helps identifying unequal variance.

• Re-examine the study design to ensure the assumption of independence is reasonable.

• Retrieve the variance inflation factor (VIF) or tolerance statistics to examine possible collinearity.

Examine potential influential point(s)

• Check statistics such as Cook's D, DFits, or DF Beta to find out if a certain data point is drastically changing your regression results. You can find more here.

Examine the change in $R^2$ and Adjusted $R^2$ statistics

• Being the ratio of regression sum of squares to total sum of squares, $R^2$ can tell you how many % of variability in your dependent variable are explained by the model.
• Adjusted $R^2$ can be used to check if the extra sum of squares brought about my the additional predictor(s) is really worth the degrees of freedom they'll take.

Check necessary interaction

• If there is a main independent predictor, before you make any interpretation of its independent effect, check if it is interacting with other independent variables. Interaction, if left unadjusted, can bias your estimate.

Apply your model to another data set and check its performance

• You can also apply the regression formula to other separate data and see how well it predicts. Graph like scatter plot and statistics like % difference from the observed value can serve as a good start.

Answer 2

I like to cross-validate my regression models to see how well they generalize to new data. My metric of choice is mean absolute error on the cross-validated data, but root mean squared error is more common and equally useful.

I don't find R2 to be a good metric of how well your model fits the training data, as almost any error metric calculated on the training data will be prone to over fitting. If you must calculate R2 on the training set, I suggest using adjusted R2.

Answer 3

You can use $R^2$ to examine how well your model fits the training data. This will tell you what percentage of the variance in the data are explained by the model.

I suggest using RMSE (root mean square error) of your predictions on your test set when compared to the actual value. This is a standard method of reporting prediction error of a continuous variable.

Answer 4

I am used to check the functional form of my parameter estimator by plotting a non-parametric (e.g. a kernel regression) or semi-parametric estimation and comparing it to the parametric fitted curve. I think this is in the first step often faster (and perhaps more insightful) than including interaction terms or higher-orders terms.

The R package np provides many nice non-parametric and semi-parametric functions, and its Vignette is well written: http://cran.r-project.org/web/packages/np/vignettes/np.pdf