I am new to the subject and I encountered this question in my text book:
Suppose that the number of features is greater than the number of samples. e.g: $𝑦_1 = b_1𝑋_{11} + b_2𝑋_{12} + b_3𝑋_{13}$
$𝑦_2 = b_1𝑋_{21} + b_2𝑋_{22} + b_3𝑋_{23}$
Why there are some linear models that achieve $\textrm{RSS}=0$? If we use only $X_1$ what can we know about the LSE? Does it equal $b_1$?
Why there are some linear models that achieve $RSS=0$?
This is straight up a fact from linear algebra, that when you have fewer equations than unknowns the system is underdetermined. (In this context, you have 2 equations and 3 unknowns, the $b_i$ variables). An underdetermined system can be solved exactly and so $RSS=0$. Contrast this with an overdetermined system which is the usual application of linear regression, where you have many more equations (datapoints) than unknowns.
If we use only $X_1$ what can we know about the LSE? Does it equal $b_1$?
Suppose you use only $X_1$. In that case your set of equations would be:
$Y_1 = b X_1$
$Y_2 = b X_2$
This is an overdetermined system. In geneal you can't solve it exactly (and so $RSS > 0$), except in the special case where $(X_1, Y_1)$ is a multiple of $(X_2, Y_2)$ in which case again $RSS=0$.
Let me just reinforce this: The regime that they're asking you about is almost always an edge case in the context of using linear regression in "day to day data analysis". The normal regime is that you have many more datapoints than unknowns and $RSS>0$.
