Suppose we deal with the linear regression model $Y=X\beta+\epsilon$, where $X$ is determined matrix, $\beta$ - the vector of coefficents, $\epsilon$ - the vector of errors.
I often meet the statement that the coefficient of determination increases as the number of regressors does. How can it be explained? Obviously, $TSS$ does not change (observed values do not change). How can I prove that $ESS$ decreases?
Thanks in advance!
When the regression coefficients are solved using ordinary least squares, it is actually minimizing the sum of squared errors ($ESS$). So if a new regressor is added and might potentially increase $ESS$, then $OLS$ would just set its coefficient to zero instead. The coefficients of the original variables would stay the same and $ESS$ would also stay the same.
In most cases, even if the new variable is just random noise, it is never perfect random noise and a positive or negative coefficient will help lower ESS even if just a little bit.
