Linear regression, conditional expectations and expected values

Question

Okay so just a bit hazy on a few things, any help would be much appreciated. It is my understanding that the linear regression model is predicted via a conditional expectation

$$E(Y|X)=b+Xb+e$$

1. Do we assume that both $X$ and $Y$ are random variables with some unknown probability distribution? it was my understanding that only the residuals and the estimated beta coefficients were random variables. if so, as an example, if $Y =$ obesity and $X =$ age, if we take the conditional expectation $E(Y|X=35)$ meaning, whats the expected value of being obese if the individual is $35$ across the sample, would we just take the average(arithmetic mean) of y for those observations where $X=35$? yet doesn't the expected value entail that we must multiply this by the probability of occurring ? but how in that sense to we find the probability of the $X$-value variable occurring if it represent something like age?
2. If $X$ represented something like the exchange rate, would this be classified as random? how on earth would you find the expected value of this without knowing the probability though? or would the expected value just equal the mean in the limit.
3. If we don't assume the dependent variables are themselves random variables, since we don't obverse the probability, what do we assume they are? just fixed values or something? but if this is the case, how can we condition on a non-random variable to begin with? what do we assume about the independent variables distribution?

Sorry if anything doesn't make sense or is obvious to anyone.

Answer 1

In the probability model underlying linear regression, X and Y are random variables.

if so, as an example, if Y = obesity and X = age, if we take the conditional expectation E(Y|X=35) meaning, whats the expected value of being obese if the individual is 35 across the sample, would we just take the average(arithmetic mean) of y for those observations where X=35?

That's right. In general, you cannot expect that you will have enough data at each specific value of X, or it may be impossible to do so if X can take a continuous range of values. But conceptually, this is correct.

yet doesn't the expected value entail that we must multiply this by the probability of occurring ?

This is the difference between the unconditional expectation $E[Y]$ and the conditional expectation $E[Y \mid X = x]$. The relationship between them is

$$ E[Y] = \sum_x E[Y \mid X = x] Pr[X = x] $$

which is the law of total expectation.

but how in that sense to we find the probability of the X-value variable occurring if it represent something like age?

Generally you don't in linear regression. Since we are attempting to determine $E[Y \mid X]$, we don't need to know $Pr[X = x]$.

If we don't assume the independent variables are themselves random variables, since we don't obverse the probability, what do we assume they are? just fixed values or something?

We do assume that Y is a random variable. One way to think about linear regression is as a probability model for $Y$

$$ Y \sim X \beta + N(0, \sigma) $$

Which says that, once you know the value of X, the random variation in Y is confined to the summand $N(0, \sigma)$.

Answer 2

There will be a LOT of answers to this question, but I still want to add one since you made some interesting points. For simplicity I only consider the simple linear model.

   It is my understanding that the linear regression model
   is predicted via a conditional expectation E(Y|X)=b+Xb+e

The fundamental equation of a simple linear regression analysis is: $$\mathbb E(Y\,|\,X) = \beta_0 +\beta_1X,$$ This equation meaning is that the average value of $Y$ is linear on the values of $X$. One can also notice that the expected value is also linear on the parameters $\beta_0$ and $\beta_1$, which is why the model is called linear. This fundamental equation can be rewritten as: $$Y = \beta_0+\beta_1X+\epsilon,$$ where $\epsilon$ is a random variable with mean zero: $\mathbb E(\epsilon) = 0$

Do we assume that both X and Y are Random variables with some unknown 
probability distribution? ... If we don't assume the independent variables 
are themselves random 

The independent variable $X$ can be random or fixed. The dependent variable $Y$ is ALWAYS random.

Usually one assumes that $\{X_1,...,X_n\}$ are fixed numbers. This is because regression analysis was developed and is vastly applied in the context of designed experiments, where the $X$'s values are previously fixed.

The formulas for the least squares estimates of $\beta_0$ and $\beta_1$ are the same even if the $X$'s are assumed random, but the distribution of these estimates will generally not be the same compared to the situation with fixed $X$'s.

if we take the conditional expectation E(Y|X=35) ... would we just take 
the average(arithmetic mean) of y for those observations where X=35?

In the simple linear model you can build a estimate $\hat\varphi(x)$ of $\mathbb E(Y|X = x)$ based on the estimates of $\hat \beta_0$ and $\hat \beta_1$, namely: $$\hat\varphi(x) = \hat\beta_0+\hat\beta_1x$$
The conditional mean least squared estimator has expression equal to the one you described if your model treats the different weights as levels of a single factor. Those models are also known as one-way ANOVA, which is a particular case of (not simple) linear model.