Linear regression and interpretation of random variables

Question

in my statistics course, it was mentioned that for the Ordinary least squares method (OLS) taking variables $X_i$ and $Y_i$ as the response variable, the random variable is solely $Y_i$.

However for the Orthogonal regression method (ORM) both the $X_i$ and $Y_i$ are considered to be random variables.

From my understanding in the OLS we optimize the regression based on individually picked values, so for each $x_i$. However, in the ORM the optimization is not made for some fixed $x_i$ value. Is this the correct reason, or is there something else?

Answer 1

It's not true that, with OLS, only the dependent variable, $Y$, is random. In fact, both can be random variables, however OLS is centered on minimizing the mean squared deviation between $Y$ and $X\beta$, conditioned on the variable $X$. That is, if your equation is $$Y = X\beta + e$$ where $e$ is a vector of IID components, then the least squares problem is to find the vector $\beta$ such that $$\mathbb{E}[(Y- X\beta)^2\mid X]$$ is minimized.

In this sense, we can often "treat" the random variable $X$ as being "non-random", since we can pull it out of the operator $\mathbb{E}[ \cdot \mid X]$, however it still can be a random variable.

ORM has to do with dealing with cases where both $X$ and $Y$ contain measurment error. That is, when $X = X + \epsilon$ and $Y = y + \mu$ where $x,y$ are unknown fixed constants and $\epsilon, \mu$ are independent random variables.