Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I read multiple times that the it is crucial in regression to (1) look at the "mean structure", and (2) at the "mean/variance relationship".
Alas, I have absolutely no idea what that means, and how it is connected to the assumptions of various regression (e.g. homoscedasticity and normal distribution in OLS, proportional odds assumption in ordered regression, etc.)
Could you hint me at some readings?
Consider the assumptions of the standard linear model:
$E[Y_i] = \beta_0 + \beta_1 X_i.$
The "errors" $\{Y_i - (\beta_0+\beta_1 X_i)\}$ are independently and identically distributed.
The first assumption concerns the "mean structure": it stipulates that the expectations of the response variables (their "means") depend linearly on the explanatory variables. Failure (or violation) of this assumption is reflected in lack of goodness of fit and, often, heteroscedastic residuals. It might be cured by introducing more variables or interaction terms in the model and/or by nonlinear re-expressions of either the independent or explanatory variables.
For more information about the mean structure, read about goodness of fit testing, introducing interaction terms, and checking for linearity of responses.
The second assumption concerns the errors and comprises two related ideas: of independence and of identical distribution. It is natural to turn to the second moments of the multivariate distribution of $(Y_i)$ to describe departures from (or violations) of these assumptions:
Lack of independence can be revealed by nonzero covariances between $Y_i$ and $Y_j$, $i \ne j$.
Lack of identical distributions can be revealed by differing variances among the $Y_i$ (lack of homoscedasticity).
(Subtracting $\beta_0+\beta_1 X_i$ from $Y_i$ does not change the variances or covariances because these are understood to be conditional on the $X_i.$ That explains why we can focus attention on the properties of the $Y_i$ instead of the properties of the errors.)
We might collectively term this the "variance structure." In the second case, frequently a departure from identical distributions attains a simple form: the variance of $Y_i$ is seen to have some definite relationship to the expectation of $Y_i$. This is the "mean/variance" relationship. Such relationships occur in actual data and often indicate that some nonlinear re-expression of $Y$ would be useful. For instance, when the variance of $Y$ is proportional to the expectation of $Y$, we might have better success fitting a linear model in which the square root of $Y$ is the explanatory variable rather than $Y$ itself.
There are many approaches to handling violations of the second assumption, ranging from generalized linear models to methods of re-expressing the variables, including Box-Cox transformations, as well as specialized models such as GARCH (for time series). Standard regression diagnostics, including plots of residuals against fitted values, aim at detecting and quantifying departures from this assumption.
Note how these considerations are related to building an appropriate model. They do not refer to the typical sizes of the residuals (the differences between the observations of the $Y_i$ and their estimated values): that is a property of the data which is beyond the control of the analyst. The difference between a linear model satisfying (1) and (2) with small residuals and another linear model satisfying (1) and (2) with large residuals is illustrated in Erogol's answer.
Mean and variance relation shows the spread of the data points on feature space. More variance means more spread on the space thus it is more likely to have bad performed regression if you are using a linear regression algorithm.
As you can see from the figures in high variance case your regression solution is rather bad performance for new comer instances even for the train set as well
