1
$\begingroup$

I'm a agronomy student in Colombia and I've been recently studying from the book Generalized Linear Models With Examples in R by Dunn and Smyth. As you can imagine, I do not have a pretty good knowledge about the underlying theory of stats.

In the book, in the second chpater, the authors propose the general form of linear regression models like this:

General Form LM

And then they say that "where $E[y_i] = \mu_i,$ and the prior weights $w_i$ are known." $E[y_i] = \mu_i$ is the expected value.

I would be really happy is someone could explain me why or how we suppose we know that value, the expected value, when we are trying to run a linear regression.

I'm sorry if I'm not being clear enough about what I'm trying to ask. Best regards,

Rafael

$\endgroup$
3
  • $\begingroup$ You are setting up a model relating the $y_i$ to the $x_{ji},$ based on unknown constants $\beta_0, \dots, \beta_p$ and $\sigma^2.$ The regression procedure will provide estimates of these unknown 'parameters' based on data $y_i$ and $x_{ji}.$ Somehow you're supposed to be able to make educated guesses about the weights $w_i.$ $\endgroup$
    – BruceET
    Aug 26, 2020 at 19:12
  • 1
    $\begingroup$ Hello BruceET. Thanks for adding the right symbols for thoso :) This is my first answer so I forgot them. $\endgroup$ Aug 26, 2020 at 19:27
  • $\begingroup$ We don't suppose that we know the value of $\mu_i$, only that it is a linear function of $x$'s. We know the $x$'s, $y$ and the weight-vector, $w$. The population parameters are all unknown. $\endgroup$
    – Glen_b
    Aug 26, 2020 at 23:57

1 Answer 1

1
$\begingroup$

Both of the instances of the authors referring to known quantities are assumptions that are necessary for standard linear regression to maintain its standard properties.

  1. The positive weights are known. Typically a regression book would say that standard OLS assumes that the error is homoskedastic, that is that the idiosyncratic variance of each observation is the same for all observations. In the authors notation this would be:

$$Var(y_i) = \sigma^2$$, but I would write it as: $$Var(y_i|x_i) = \sigma^2$$ to stress the fact that the idiosyncratic error is condition on the data. It is the variation no due to the covariates.

So this is the normal way that this is presented. Then typically a book will say that often this assumption is violated and that the idiosyncratic error may be more complicated, such as being heteroskedastic (each observation has it's own variance $Va(y_i|x_i) = \sigma_i^2$) or autocorrelation (the errors are correlated amoungst each other, common in time series). There are modifications to the model such as weighted least squares or feasible weighted least squares or modifications to the way we calculate the standard errors such as heteroskedasticity robust standard errors which can deal with this.

In the book you are following, they point out that you can still more or less use ordinary least squares if there is heteroskedasticity of the form $Var(y_i|x_i) = \sigma_i^2 = \sigma^2/w_i$ and for some reason you know what the weights $w_i$ are for all $i$. In practice most of the time you would not know this, but what it means in loose terms is you know which observations are noisier or less noisy than others and can quantify that in terms of the weight $w_i$.

The way that this would work is by running the regression of $\frac{y_i}{\sqrt(w_i)}$ on $\frac{x_{i,1}}{\sqrt(w_i)}, \frac{x_{i,2}}{\sqrt(w_i)}, \dots, \frac{x_{i,p}}{\sqrt(w_i)}$ and an intercept. If $w_i$ is large, you are effectively downweighting the influence of that observation because it is noisy. If $w_i$ is small you are upweighting it because it is giving you lots of information. Again, these are just assumptions and as I mentioned there are ways to weaken these assumptions if the analyst feels they are too strong.

  1. Where $E[\mu_i]$ is known.

This is again an assumption. One way to think about linear regression is as specifying a model for the conditional expectation. Again, it is more common and my personal preference to express this as a conditional expectation:

$E[\mu_i|x_i] = E[y_i|x_i] = \beta_0 + \sum_{i=1}^px_i\beta_i$

The idea is that in order to recover the true conditional expectation, it needs to be linear (in the coefficients) of the model. In practice, do we usually know that this is true. No not typically, it is an assumption. If you go to chapter 2.3 of the text you reference, they show examples where the assumptions are violated. It is usually easy to check that assumptions are violated when they are violated grossly, but we can never fully verify they are satisfied without some outside knowledge coming outside of the data.

This is beyond the scope of this answer, but linear combination of variables have nice properties that still may justify them even when the assumption is not quite true. Sometimes we can think of a linear regression as a taylor expansion or local approximation to the true conditional expectation. By including things like higher order terms $x^2, x^3$ etc or other basis expansions (or things like splines) these approximations can become more accurate (in terms of ability to predict the outcome in or out of sample) or plausible. Such an approximative model will not necessarily be unbiased or enjoy some of the efficiency properties that OLS can have, but can still be quite useful. This is often how people think of linear regression in practice anyway, particularly in industry.

The answer to how do we know is extremely case by case. What do you know about the variables. As an agronomist you may sometimes be able to look to other studies or theories about how crops behave to partially justify the assumptions you make in a particular model. The art of statistics is about matching plausible assumptions about the real world that produces the data with models. Understanding the assumptions of a model, how they can or cannot be weakened or strengthened, and when they plausibly hold is the entire battle of an applied statistician or data scientist.

$\endgroup$
5
  • $\begingroup$ Thanks for the making this clearer to me. So, the assumption that tells us that we know the expected value of some data set is more about knowing that our model is linear in the paremeters than about estimating an exact value for that expected value? I am so sorry if I am not getting the idea right but this concept of expect value is still somehow confusing for me. $\endgroup$ Aug 26, 2020 at 22:23
  • $\begingroup$ The way to think about it is if it is impossible to write the conditional expectation in terms of a function that is linear in parameters then it will be impossible for OLS to recover the true conditional expectation (except maybe in the limit). When we run a particular regression, say $Y \sim X_1 + X_2$ we are implictly modelling the conditional expectation of Y given X_1, X_2 as $E[Y|X_1,X_2] = \beta_0 +X_1\beta_1 + X_2\beta_2$, where we estimate the betas to make this fit as close as possible. If the real conditional expectation isn't linear or close to it then... $\endgroup$ Aug 27, 2020 at 1:29
  • $\begingroup$ The regression won't fit as well as it could and the intepretation as a conditional expectation may not make much sense. Of course one can still use OLS to fit conditional expectations like say $E[Y|X_1,X_2] = \beta_0 + X_1\beta_1 + X_2\beta_2 + X_1^2\beta_3 + X_2^2\beta_4 + X_1X_2\beta_5$. This is linear in parameters. To know which makes more sense you need to make inferences about the underlying data generating process and use knowledge of what the variable represents. OLS can't handle conditional expectations like $E[Y|X_1,X_2] = X_1\beta_1 + X_2\frac{1}{\beta_1}$ for example $\endgroup$ Aug 27, 2020 at 1:38
  • $\begingroup$ Thanks you a lot. So, in conclusion, when we use OLS, which is what we use when we're doing linear regression, we ASSUME that our conditional expectation for a certain value of Y, CAN BE modelled by a linear function of X, or the X's that we're dealing with? By the way, can you rcomend me some short reading to get a better idea about how do we use conditional expectation in OLS? Again, thanks a lot Dr. Tyrel. $\endgroup$ Aug 27, 2020 at 13:59
  • $\begingroup$ Maybe this will be helpful: timlrx.com/2018/02/26/…. $\endgroup$ Aug 27, 2020 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.