Model errors, residuals and heteroscedasticity I have a quick question about the correct way to describe variance functions when seeking to cope with heteroscedasticity. As I understand it the statistical error of a model represents the departure between a sample and the true function value. In practice we don't usually know the true function value. In contrast, residuals are the difference between a sample and the estimated function value, which we do know. Now, assume that I have a model with heteroscedasticity and I want to model it as
$$\varepsilon_i \sim N(0, \sigma^2 \times X_i).$$
Would it be correct to say that the variance in the statistical errors were modeled as a function of $X$? Or, would it better to state that the variance in the residuals was modeled as a function of $X$?
 A: Conclusions
It makes no sense to model the variance of the residuals explicitly, because the variance depends on the fitting procedure, which is not part of the model.  We must model the variance of the errors and then, for our chosen fitting procedure, we may determine the distribution of the residuals.
Assumptions
The question asks how to interpret "$\varepsilon_i$," but unfortunately it does not define $X$, $X_i$, or $\sigma^2$.  Nevertheless we can make some guesses that may be useful:

*

*Evidently each $\varepsilon_i$ is a random variable with a numerical (not vector) value, because it refers either to an "error" or a residual.


*Therefore "$\sigma^2 \times X_i$" must be a number.  Taking $\sigma^2$ to be a scale parameter, we conclude that the $X_i$ are numbers.


*The $X_i$ could be random variables, but if so, this is a complicated model.  For simplicity (and because it doesn't affect the concepts or nature of the subsequent analysis), let's take them to be fixed values (one for each $i$).  So, let the $X_i$ be either independent values or covariates, treated as known and measured without appreciable error.
Restatement of the question
This situation is a general regression setting.  To be clear and specific, let the index $i$ designate observations; for each $i$, let $Z_i$ be a vector of independent values; let $g$ be a (known) real-valued function with $X_i = g(Z_i)$; let $\theta$ be a vector of (unknown) model parameters to be estimated; let $Y_i$ be the dependent values; let $\sigma$ be another (scalar) parameter, known or unknown; and let $f$ be the (known) function referred to in the question.  In these terms I guess the model in question is of the form
$$Y_i = f(Z_i, \theta) + \varepsilon_i$$
and the errors $\varepsilon_i$ are assumed to be independent, normally distributed with zero mean, and to have variance
$$Var(\varepsilon_i)=\sigma^2 g(Z_i).$$
Let $\hat{\theta}$ be any estimates of the parameters.  Specifically, $\hat{\theta}$ is some function $t$ of all the data:
$$\hat{\theta} = t(\mathbf{Z}, \mathbf{Y}) = t(\mathbf{Z}, f(\mathbf{Z}, \theta)+\mathbf{\varepsilon})$$
($\mathbf{Z}$ is the vector of $Z_i$, $\mathbf{Y}$ is the vector of $Y_i$, and $\varepsilon$ is the vector of $\varepsilon_i$).  $t$ is the estimator, often least squares or maximum likelihood.
The fitted model therefore is
$$\hat{Y} = f(Z, \hat{\theta})$$
and the residuals, by definition, are
$$e_i = \hat{Y_i} - f(Z_i, \hat{\theta}).$$
Now the question, at least as I have interpreted it, can be restated:

"Would it be correct to say that $Var(\varepsilon)$ should be modeled
as a function of $X$ or would it be better to say that $Var(e)$ should
be modeled as a function of $X$?"

Analysis
At this point it is clear that the $\varepsilon_i$ have nothing to do with the fitting procedure or the parameter estimates $\hat{\theta}$.  Recalling that $\hat{\theta}$ is determined by the estimation procedure $t$ and that $t$ is deterministic (it's a specific function of its arguments), we note that the $e_i$ are random only insofar as $t$ depends on the errors $\varepsilon_i$.  Therefore their variances, $Var(e_i)$, depend (in a potentially complicated way) on $t$ itself. Trying to model $Var(e)$ would inextricably link the underlying model (an approximate description of reality) with the parameter estimation procedure, which makes little sense and could only be counterproductive.
A: A generative model for heteroskedastic regression would be to say that the responses were drawn from a normal distribution where the mean and variance are functions of the explanatory variables, i.e.
$y_i \sim N(f(x_i;\beta_\mu), g(x_i;\beta_\sigma))$
where $\beta_\mu$ and $\beta_\sigma$ are the parameters of the two component models, and are generally jointly optimised by minimising the negative log-likelihood. The functions $f(\cdot)$ and $g(\cdot)$ estimate the conditional mean and conditional variance of the target distribution; there isn't any real need to talk about "residuals" or "statistical errors".
I would say that it would be better to state that "the conditional variance of the response variable is modelled as a function of $X$".
