9
$\begingroup$

Questions:

  • Are improper linear models used in practice or are they some kind of curiosity described from time to time in scientific journals? If so, in what areas are they used?
  • Are there other examples of such models?
  • Finally, would standard errors, $p$-values, $R^2$ etc. taken from OLS for such models be correct, or should they be corrected somehow?

Background: Improper linear models are described from time to time in the literature. In general, such models can be described as

$$ y = a + b \sum_i w_i x_i + \varepsilon $$

what makes them different from regression is that $w_j$'s are not coefficients estimated in the model, but are weights that are

  • equal for each variable $w_i = 1$ (unit-weighted regression),
  • based on correlations $w_i = \rho(y, x_i)$ (Dana and Dawes, 2004),
  • chosen randomly (Dawes, 1979),
  • $-1$ for variables negatively related to $y$, $1$ for variables positively related to $y$ (Wainer, 1976).

Also it is common to use some kind of feature scaling, like converting variables into $Z$-scores. So, this kind of model can be simplified to univariate linear regression

$$ y = a + b v + \varepsilon $$

where $v = \sum w_i x$, and can be simply estimated using OLS regression.

References:
Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571-582.

Graefe, A. (2015). Improving forecasts using equally weighted predictors. Journal of Business Research, 68(8), 1792-1799.

Wainer, Howard (1976). Estimating coefficients in linear models: It don't make no nevermind. Psychological Bulletin 83(2), 213.

Dana, J. and Dawes, R.M. (2004). The Superiority of Simple Alternatives to Regression for Social Science Predictions. Journal of Educational and Behavioral Statistics, 29(3), 317-331.

| cite | improve this question | | | | |
$\endgroup$
  • 3
    $\begingroup$ In what sense would the statistics derived from these models be "incorrect"? $\endgroup$ – whuber Jul 19 '15 at 13:19
  • 1
    $\begingroup$ When the $w_i$s are pre-specified & $b$ estimated, this is just data reduction carried out on the predictors - common enough in various forms (see e.g. the Glasgow Coma Scale & the Charlson Co-morbidity Index) - which won't affect the validity of inference in the usual OLS framework. When $y$ is used to determine the $w_i$s, the standard errors &c. will be out, in the optimistic direction I'd think. $\endgroup$ – Scortchi - Reinstate Monica Dec 8 '15 at 17:45
  • 1
    $\begingroup$ It wasn't an informed comment - the papers are still on my "to read" pile. I just wondered:-"why 'improper'?". It's not unusual for a predictor to be a linear combination of other variables - an average of several measurements, a principal component score, a prediction from another regression, the level from an exponentially smoothed time series, or a calculated value from a well-established or an ad hoc index. Not estimating the weights from the response spares degrees of freedom, helping to avoid over-fitting with smaller sample sizes. $\endgroup$ – Scortchi - Reinstate Monica Dec 9 '15 at 10:43
  • 1
    $\begingroup$ In e.g. Beddhu (2000), "A simple comorbidity scale predicts clinical outcomes and costs in dialysis patients" Am. J. Med., 108, 8 the model equation has the same form as yours where the $x_i$s are defined as the indicator variables for diabetes, lymphoma, & c., & the $w_i$s are pre-specified. I suppose what I'm saying is that the distinction between "improper" & "proper" regression models seems to rest on the notion of a God-given set of $x_i$s, for each of which a "proper" model would estimate a coefficient. $\endgroup$ – Scortchi - Reinstate Monica Dec 9 '15 at 11:35
  • 4
    $\begingroup$ When $w_i = \rho(y, x_i)$, & if $\rho$ were estimated from the same data the model is fit to, that'd be quite a different kettle of fish. $\endgroup$ – Scortchi - Reinstate Monica Dec 9 '15 at 11:44
1
$\begingroup$

In effect, it seems to me this is an assortment of assumed covariance structures. In other words, this is a type of Bayesian prior modelling.

This gains in robustness over an ordinary MLR procedure because the number of parameters ($\downarrow$df) is reduced, and introduces inaccuracy because of augmented omitted variable bias, OVB. Because of the OVB, the slope is flattened, $|\hat\beta|<|\beta|$, the coefficient of determination is reduced $\hat{R}^2<R^2$.

My personal experience is that the superior to the Bayesian approach is to use better modelling; transform parameters, use other norms, and/or use nonlinear methods. That is, once the physics of the problem and the methods are properly explored and co-ordinated, the F statistics, coefficient of determination, etc. improve rather than degrade.

| cite | improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.