# Mean structure and mean/variance relationship in regression

I read multiple times that the it is crucial in regression to (1) look at the "mean structure", and (2) at the "mean/variance relationship".

Alas, I have absolutely no idea what that means, and how it is connected to the assumptions of various regression (e.g. homoscedasticity and normal distribution in OLS, proportional odds assumption in ordered regression, etc.)

Could you hint me at some readings?

-

Consider the assumptions of the standard linear model:

1. $E[Y_i] = \beta_0 + \beta_1 X_i.$

2. The $\{Y_i\}$ are independently and identically distributed.

The first assumption concerns the "mean structure": it stipulates that the expectations of the response variables (their "means") depend linearly on the explanatory variables. Failure (or violation) of this assumption is reflected in lack of goodness of fit and, often, heteroscedastic residuals. It might be cured by introducing more variables or interaction terms in the model and/or by nonlinear re-expressions of either the independent or explanatory variables.

The second assumption concerns the "errors" and comprises two related ideas: of independence and of identical distribution. It is natural to turn to the second moments of the multivariate distribution of $(Y_i)$ to describe departures from (or violations) of these assumptions:

• Lack of independence can be revealed by nonzero covariances between $Y_i$ and $Y_j$, $i \ne j$.

• Lack of identical distributions can be revealed by differing variances among the $Y_i$ (lack of homoscedasticity).

We might collectively term this the "variance structure." In the second case, frequently a departure from identical distributions attains a simple form: the variance of $Y_i$ is seen to have some definite relationship to the expectation of $Y_i$. This is the "mean/variance" relationship. Such relationships occur in actual data and often indicate that some nonlinear re-expression of $Y$ would be useful. For instance, when the variance of $Y$ is proportional to the expectation of $Y$, we might have better success fitting a linear model in which the square root of $Y$ is the explanatory variable rather than $Y$ itself.

There are many approaches to handling violations of the second assumption, ranging from generalized linear models to methods of re-expressing the variables, including Box-Cox transformations, as well as specialized models such as GARCH (for time series). Standard regression diagnostics, including plots of residuals against fitted values, aim at detecting and quantifying departures from this assumption.

Note how these considerations are related to building an appropriate model. They do not refer to the typical sizes of the residuals (the variances of the $Y_i$): that is a property of the data which is beyond the control of the analyst. It is what it is. The difference between a linear model satisfying (1) and (2) with small residuals and another linear model satisfying (1) and (2) with large residuals is illustrated in Erogol's answer.

-