If I add an extra variable to a regression model I already have, and the R squared increases, does this usually mean the new model is better? Lets say I'm estimating a model with ordinary least squares, and that my initial model (with an $R^2$ of 0.5) is
$$
y_i = \beta_o + \beta_1x_{i1} + \beta_2x_{i2} + \varepsilon_i
$$
Lets say I add a new variable to my model, so that:
$$
y_i = \beta_o + \beta_1x_{i1} + \beta_2x_{i2} + \beta_3x_{i3} + \varepsilon_i
$$
The new model gets an $R^2$ of 0.6. Does this mean that the model is usually better, or are there any obvious caveats?
 A: The best model will describe the data as effectively as possible while being as simple as possible.
Each additional parameter you add will capture more variance in the data. This means $R^2$ will always increase as you add more parameters. You should ask yourself how valuable the increase was given the addition of the extra parameters.
Metrics like information criteria or adjusted $R^2$ help answer this. They compare how well the model fits the data with the number of parameters used. You could also consider building the model on a subset of the data, and testing it on another.
A: The obvious caveat is that $R^2$ can be driven high by modeling the noise. In the extreme, this means that you are just playing connect-the-dots. The result is that your model lacks an ability to generalize, meaning that your ability to predict in the future is minimal and that your understanding of the process that generated the observations is minimal.
This is why machine learning people like out-of-sample testing. If you get an increase in $R^2$ while also getting the $R^2$ to increase when you apply the models to out-of-sample data, you would have more confidence in the model fitting to the signal rather than the noise. Alternatives, perhaps preferable to out-of-sample testing, include metrics like adjusted $R^2$ and various forms of information criteria (which penalize a model for having high numbers of parameters, meaning that you don't just need a bigger model to improve the fit but improve the fit a lot in order for adjusted $R^2$ or information criteria to improve).
