Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am concerned about how unequipped most people are (both within and without academia) to properly employ standard model building methods such as linear regression and to interpret the results of these models. Both from my own observations and the literature, it is clear that most people are being poorly served by the standard statistical tools they have available.

To improve this situation, I would like to propose a different set of model evaluation output tables (examples shown below in Table 2 and Table 3) to replace the standard output table that are almost universally used today (an example shown in Table 1). This new output format is robust to human error (both in model specification and analysis) to a much greater extent that is our current format.

I would like ask people for their critique of this proposed method and whether there was interest in using it.

(Sorry for the length of this, it got much longer than expected)

Approach Overview

First, calculate an "honest $R^2$" using leave one out cross validation (LOOCV) to get an overall measure of the model's fit. Simply put, remove each data point in turn, estimate the model without that data point, then use the reduced model to calculate the value of the removed data point and use these errors to estimate your $R^2$.

Then for each coefficient in the model, remove that parameter from the model and calculate an "honest $R^2$" for that submodel. Calculate the effect of the adding the coefficient to the model as the overall model's $R^2$ minus the $R^2$ of the submodel when the coefficient is removed.

Thus a positive value for this coefficient $R^2$ indicates the fit improves when it is added to the model; a negative value indicates the fit decreases.

Results would be reported as a simple table of honest $R^2$ values (see examples in Table 2 and Table 3 below).

Discussion

This approach is designed to be straightforward and to do three things:

  1. Provide end users with the information they need to interpret the (practical) significance of models and coefficients
  2. Provide end users with this information in a form they are already familiar with
  3. Minimize the effect of user error both in developing the models and interpreting results

I think this approach does all three. The use of the universally understood $R^2$ metric should mean it is readily understandable and can plug right into users' existing mental frameworks.

Second, the use of this approach is robust: both to misspecified models and to poor interpretation. Most models depend on certain assumptions that users often fail to check are met. Of course, p-values for coefficients in linear regressions depend on certain parametric assumptions that are generally violated to a greater or a lesser extent. This proposed approach is resilient to such issues (the main issue it is susceptible to is, as with regular output statistics, correlation between observations, which could lead to an overestimation of the honest $R^2$).

In regards to the interpretation of results, p-values or often misinterpreted (as has been extensively noted). I believe this honest $R^2$ approach is much more resilient to these issues as it is focused on effect size (which is often what people incorrectly take the p-value as a proxy for). It emphasizes practical significance in place of statistical significance which is a concept people have a lot of trouble with.

Furthermore, this method allows the direct comparison of results to those generated by other methods (even those that don't generate likelihoods) such as machine learning techniques such as random forests.

One issue with this approach is computational burden. Where LOOCV is computationally infeasible I would suggest 10- or 5-fold CV. Since these methods should result in higher error (as the model is training on smaller data sets), the honest $R^2$ values reported by them would be conservative. Therefore they can be used and reported in placed of the LOOCV with any risk of mis-comparison being of a conservative nature.

Another issue is the LOOCV estimate is known to have high bias. I'm not really sure how this could be dealt with or if it is a serious problem. One last point is the LOOCV is asymptotically equivalent to AIC, so this could fit into that paradigm a bit.

Applied Example

Taking a set of housing data ( http://archive.ics.uci.edu/ml/datasets/Housing ) we try to predict average house value in a suburb based on average house age, number of rooms in the house, pollutant levels, pupil to teacher ratio, and proximity to a highway. We start with a linear regression.

Below is similar to what most software will currently output (this specific table was generated by R).

Table 1. Standard regression table.

Coefficients:
               Estimate Std. Error t value Pr(|t|)    
(Intercept)     7.76739    4.98881   1.557 0.120112    
AGE            -0.01509    0.01378  -1.096 0.273773    
ROOMS           7.00565    0.41172  17.015    2e-16 ***
NOX           -13.31418    3.90262  -3.412 0.000698 ***
PUPIL.TEACHER  -1.11645    0.14799  -7.544 2.17e-13 ***
HIGHWAY        -0.02487    0.04257  -0.584 0.559341    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 5.819 on 500 degrees of freedom
Multiple R-squared: 0.6037, Adjusted R-squared: 0.5997 

The more honest statistics results are shown below. Note that these results aren't surprising given what we saw in table 1. The parameters that were not significant in Table 1, result in worse models as measured by the change in the honest $R^2$ (again the coefficient Honest $R^2$ value indicate how the honest $R^2$ changed when that coefficient was added to the model).

Table 2. Proposed "honest" statistics table.

          Item Coefficient Honest.R2  
  -Full Model-              0.593080  
   (Intercept)      7.7674 -0.000998  
           AGE     -0.0151 -0.000409
         ROOMS      7.0056  0.228901
           NOX    -13.3142  0.008123
 PUPIL.TEACHER     -1.1165  0.045506
       HIGHWAY     -0.0249 -0.002018

Finally, this approach allows us to directly compare more exotic algorithms such as Random Forests. Unlike Bayes factor, AIC, BIC or some other methods, likelihoods are not required for comparison. Anything that creates predictions can be compared. Classification algorithms can also fit into this scheme if you use one of the common pseudo $R^2$ approaches for calculating $R^2$.

Table 3. Comparison between algorithms.

(In this case Support Vector Machines and Random Forests)

        SVM               |         Random Forest
          Item Honest.R2  |             Item Honest.R2
  -Full Model-  0.71052   |     -Full Model-   0.7450
           AGE  0.00834   |              AGE  -0.0128
         ROOMS  0.34722   |            ROOMS   0.2068
           NOX  0.02338   |              NOX   0.0654
 PUPIL.TEACHER  0.01813   |    PUPIL.TEACHER  -0.0127
       HIGHWAY -0.00381   |          HIGHWAY  -0.0255

Side note: It is interesting to interpret these results as we can see in the linear model pupil per teacher has a statistically significant effect on housing prices. While in the random forest model, it does not. Since the random forest model is the most predictive of the models, I would have to conclude that this was evidence that this coefficient did not significantly (practical significance) affected housing prices. This is a small illustration of the weakness of blindly applying linear models to everything and using that to carry out hypothesis testing.

Implementation

I have uploaded (rough-hewn, unoptimized) code in R to implement this technique at:

http://dl.dropbox.com/u/94002/HonestStats.R

If there is interest, I can optimize this to be much faster for linear models and implement things like k-Fold CV.

share|improve this question
    
Regardless of content, this is an advertisement not a question. Recommend you write up the idea and bundle the functions into an R package. –  conjugateprior Mar 4 '12 at 12:42
    
To add to @Conjugate's comment, I think this might easily be turned as a question about the added value of a summary statistic of model predictive accuracy. As far as R is concerned, I should note, however, that there exists such cross-validated measures of predictive accuracy in the linear case (DAAG::cv.lm, for example), as well as bootstrapped measures of predictor importance, e.g. relaimpo. Moreover, the caret package offers very flexible solution to model selection. What are your thoughts? –  chl Mar 6 '12 at 23:30

4 Answers 4

In addition to Michelle's answer, predictive ability (as measured by $R^2$) is not relevant to all uses of regression.

In your example, if one is interested in the difference in mean houses prices, comparing houses whose NOx level differs by one unit but that have identical values of all the other covariates, then (given some assumptions of linearity) the regression in Table 1 is the right one to do, regardless of its $R^2$.

In my experience, getting people to translate between "what quantity is of interest?" and "what regression do we do?" is much more challenging than quantifying predictive ability.

share|improve this answer
    
I think this approach does get at what you are looking at (the model selection aspect is really just a bonus). For NOX instead of just being able to say one unit increase drops housing prices by 13 units; you can use Table 2 to say that and that NOX explains about 1% of the total home price (as measured by the reduction in squared error over the simpler model). –  Scott Mar 6 '12 at 15:14
    
The idea is that one may want to use Table 2 (instead of similar output from a regression with fewer/more variables) based on criteria that have very little to do with prediction. The original question suggests ways to help people interpret their regression output; I suggest that focusing on prediction and $R^2$ may not help them do that. NB your description of the NOx effect is strongly causal, which may not be warranted. –  guest Mar 7 '12 at 0:09

I disagree that another overall method evaluation is needed, what perhaps would be more useful is if people reported model summaries using the statistics already available to them, for example:

I see no need for another statistic. The misinterpretation of p-values is, I think, a different problem.

Update based on comment below: Your method appears to ignore how models account for variance. When you have the "missing variable" problem, some of the variance accounted for by the absent variable is assigned to variables correlated with it that are included, and the rest is error. This happens because of measurement error, use of proxy variables (particularly in social sciences), etc. So when you add a new variable to the model, you're not getting a pure adjustment to variance explained. The situation is even more complicated if you bring in interactions. Essentially your "honest R^2" advocating a type of step-wise model (where the added variable is added in at the last step), and there are major issues with those models.

share|improve this answer
    
As I see it the issue with Mallow's CP, AIC, and BIC is primarily that they are unfamiliar to most end users (again, this is not targeted at the statistical community). You are asking people to use something new and strange, and for the most part they simply won't understand. There is also the issue of comparing models between different application in which cases something like AIC is useless. Adjusted $R^2$ will under-penalize additional parameters so has over-fitting issues (so does Honest $R^2$ but to a much lesser extent). –  Scott Mar 6 '12 at 15:05

What you described is pretty much already captured within Stepwise Regression methodology with Hold Out periods. At least as presented by a software such as XLStat, the latter selects the best variables and discloses what the R Square is for the best one variable model, best 2 variable model, etc... It keeps on adding variables until the adjusted R Square value does not rise much anymore. So, you can readily see how much incremental information each additional variable provides.

share|improve this answer

The consensus thus far seems to be against, and I have to say that I agree. Every so often, I come across the argument 'people make so many mistakes with stats, all these problems would go away if we just switched to _________'. I have never yet found that convincing, and I'm afraid don't here either. I recognize that there are some merits to the approach you favor, but it doesn't address all the problems that can occur with the old methods, would almost certainly be misused just as readily as old approaches, and isn't necessarily better overall than current methods when they are used well by someone who knows what they are doing and cares about getting it right.

I rambled on interminably about a similar issue recently here--it's not exactly the same, but it's similar enough to get the idea. My main point is that much of the poor data analysis that occurs can be best described as 'mechanical' (or 'rote', in Cleveland's terms). Switching one mechanical approach for another is unlikely to change that. I can think of two related issues, which I didn't discuss there:

  • There are a lot of people who need to be able to analyze data in their work who have weak mathematical backgrounds and/or are math-phobic. We are free to dislike this aspect of reality and to grouse about it, but it isn't going away. Part of the reason for poor analyses in practice is that a lot of people don't really understand what is going on, and see statistical analyses as though they work somehow by magic. To reduce problematic analyses we need to find ways to get people to a basic (non-magic based) conceptual grasp of how statistical analyses work. (It should be noted that sites like CV are part of the answer to this.)
  • There are a lot of smart, interested people who are absorbed by the content-matter that they deal with, but who have little interest in the methodology (statistical or otherwise) with which that subject-matter information is intertwined. We need to get people on board with the fact that methodological issues are very important as well. This is not easy to do. You can point out actual cases where things were done incorrectly and important conclusions were missed or gotten wrong, but this can easily come off as overbearing and turn people off. I don't have a good solution.

Better data analyses will emerge all by themselves (without any new techniques), when these three problems are solved. That is, when people think the methodological issues are important, understand them reasonably well, and don't simply apply them mechanically. So long as those conditions continue, however, new techniques will not guarantee better analyses. You make the case that your approach will help with #2, and it very well might, but that's still not enough.

Regarding the merits of your approach, I think it has some strengths and weaknesses. Others have pointed to some issues with which I agree and so won't repeat them. However, I do want to say that using predictive accuracy makes this approach more appropriate for predictive modeling and less applicable to explanatory modeling (see here for a discussion on CV, and see the links listed on that page). Lastly, on a stylistic note, I would recommend you change the signs in your tables (2 & 3) so that the numbers represent the change from the top line -Full Model- Honest.R2 if that term is dropped. This will be much clearer and more intuitive for people.

I'm afraid this response comes off pretty negative; I don't mean to be. Your approach clearly has some good ideas. I simply do not think that it, or any other candidate solution of its type, will solve our problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.