3
$\begingroup$

I am talking about a situation in which I have several continuous predictor variables predicting a continuous outcome. One of the predictors has a very non-normal distribution and has some wild outliers.

I intend the generalize the regression model to a wider population. Does the non-normality and/or the presence of outliers preclude my analysis, or cause difficulties in interpretation?

What should I do?

$\endgroup$
1

3 Answers 3

4
$\begingroup$

This post poses two different questions, but since both are related to a single regressor, lets treat them as one.

Distribution of regressors
Consider the usual multiple linear regression specification

$$y_i = \beta_0 + \beta_1x_{1i} +...+\beta_kx_{ki} + u_i$$

If regressors are stochastic then the distribution of $y_i$ results from the combination of the regressor's distribution and the error distribution. In the special case where all regressors follow a normal distribution, and the error term is also assumed normally distributed too, then we can conclude that the distribution of $y_i$ should also be normal. But this is a very special case. In general, recovering the distribution of $y_i$ is infeasible...
We side-step the matter by considering the distribution of $y_i$ conditional on the regressors. Our inference and conclusions from the model are conditional on the regressors (not only on the specific sample we have available, but in general on the regressors as random variables and on their joint sigma-algebra).
The stochastic assumptions are stated in terms of this conditionality: the error term has a zero-mean conditional on the regressors, its variance is constant conditional on the regressors. Exact test results are based on the assumption that the error term is distributed normally conditional on the regressors, etc.
All modern treatments of least-squares estimation do that. Maximum likelihood is mostly implemented as conditional maximum likelihood, although we usually don't say it explicitly: when we say "the joint density of the sample" in most cases, the term "joint" refers to the fact that we consider many observations, not to the joint distribution of the dependent variable and the regressors.

Theoretically, perhaps we would want to know the unconditional distribution of the dependent variable, so that we could handle the uncertainty surrounding it without the need to have specific information about its associations/covariance/causal relations with other variables... but if information is available, why not use it? Conditional inference is more focused (and so less uncertain) than unconditional inference but if it is mistaken (misspecification issues), it creates larger inaccuracies... This is the trade-off we have to live with. So don't worry about the non-normality of your regressor, it is not an issue.

Outliers
Outliers on the other hand is an issue. Many statistics (like the sample mean or the sample standard deviation) are not robust to the existence of outliers (while the median and the mean absolute deviation are, comparatively). There is a tendency to just delete outliers from the sample, based on the argument that they do not reflect "usual behavior" and so they are misleading -but not all situations validate such an argument and action.

There is a large literature regarding the issue. For example, look up "Influential Observations Analysis" where various measures have been constructed that go one step further regrading outliers and evaluate what degree of influence has an outlier on the estimation results that we have obtained from the model. I would say such a step is essential before making any decision about whether to delete or smooth outliers, or not, or factoring them in, when interpreting the results, or not.

$\endgroup$
4
  • 3
    $\begingroup$ The fact that "outliers" can damage regression fits doesn't imply that the modeling process is dependent on distributions for $X$. I think of high leverage points as being damaging as a consequence of most of our model fitting methods not being robust. To handle heavy right-tailed variables I often fit restricted cubic splines in the cube root of the predictor. $\endgroup$ Apr 19, 2014 at 14:23
  • 1
    $\begingroup$ @Frank Harrell I don't see where in my answer I say or imply that the modeling process is dependent on the distribution of $\mathbf X$, on the contrary. $\endgroup$ Apr 19, 2014 at 17:30
  • $\begingroup$ Sorry my comment was in the wrong location. It was meant for the original poster. $\endgroup$ Apr 19, 2014 at 20:45
  • $\begingroup$ @Frank Harrell Oh, ok.Why don't you turn it into an answer? It provides specific suggestions, which is good. $\endgroup$ Apr 19, 2014 at 21:11
2
$\begingroup$

In OLS the distribution of the predictors doesn't matter. Your predictors form a design matrix $X_{ij}$, where $j$ is the predictor, and $i$ is the number of the observation, so the model specification is $Y_i=\sum_jX_{ij}\beta_j+\varepsilon_i$ or $Y=X\beta+\varepsilon$ in vector notation. The only distribution that matters here is of the error term $\varepsilon$.

Outliers is an interesting topic. In OLS we assume that $X$ are known and their distribution does't matter, so the outliers are not a concern at all so long you know the value of $X$.

The outliers in OLS come from the error terms. The distribution of $\varepsilon$ matters in several ways, and the outliers are important.

So, my answer is that distribution and outliers in the predictors is not a problem.

However, by the way you asked the question I can suspect that you have a problem, which you did not formulate accurately. My guess is that you have a situation where $Y$ depends on a true state, which is not observed directly or observed with an error. So that you see the predictors, but they are not true states of the system.

$\endgroup$
1
$\begingroup$

The distribution of the predictors does not matter at all. This is a quite common misunderstanding. What rather matters is the distribution of the errors in the fitted model (plus other important assumptions). See this blogspot by Andrew Gelman & the discussion in the comments, and also this previous question.

$\endgroup$
2
  • $\begingroup$ Outliers in the input space is a major issue in most types of models. So, I do not fully agree with your answer $\endgroup$
    – Michael M
    Apr 19, 2014 at 12:31
  • $\begingroup$ The links I provided and the other answers and comments here clearly state that the distribution of X does not matter per se. If you can convincingly argue otherwise, you should post it as an answer. $\endgroup$ Apr 20, 2014 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.