So I am looking into a regression model that is supposed to predict the value of a house based on numerous independent variables. What I don't quite understand is how to select the "best" model when eliminating insignificant variables. The original model contained four independant variables of which three turned out to be insignificant (p-value > 0.05). So I removed, say X1, which had the largest p-value (small t-stat). The new model containing three variables clearly appeared to be better, as Adjusted R2 rose (I understand Adjusted R Square is suitable for comparing models with different number of parameters). The F-tests also seemed to confirm that removing X1 was the right thing to do due to its insignificance. However, in the three variable model there was still one insignificant variable, X3. So I removed it as well. Now, in the two variable model, both variables were reliable (small p-values), but Adjusted R Square was actually somewhat lower than that of the previous model. F-tests seemed to indicate that X3 had a small enough impact on the model to be eliminated. Which is actually the better model statistically and economically? The three variable model with the highest Adjusted R Square or the two variable model with no insignificant variables but lower Adjusted R2? I couldn't quite figure out the logic behind this.
If your goal is prediction rather than trying to infer causality, there is no need to remove any predictor variables, as @Michael Mayer has already commented, unless there is a significant cost to obtaining the values of some of them. And although your predictor variables may be the "independent" variables in your analysis, it's likely that many are correlated to each other (e.g., number of bedrooms, number of bathrooms, size of lot). In those cases trying to remove variables that are "insignificant" in analysis of a particular sample may lead to weird, counter-intuitive results that do not generalize well for future predictions.
If you do need to remove predictor variables for some reason, follow specific defined methods for hierarchical/stepwise analyses as suggested by @jmbejara, rather than trying to make up the rules as you go. These methods use better ways to compare models than the adjusted R-squared values, and they are available in R and other statistical analysis software.
From what I can tell, it generally depends on your purpose. If you are trying infer causality, then you need choose variables that specify some structure that actually has some theory behind it. Do you actually have good, intuitive reasons for including one variable or the other? (However, this is useless without some sort of identification strategy to disentangle other effects and biases).
On the other hand, if your goal is only to create a model with predictive power, it seems like in general you want to choose variables that make significant contributions to the fit of the model. For prediction and descriptive analysis, it seems to me like different fields have different rules. See this discussion about hierarchical and stepwise regressions of an example of different kinds of rules. In the case you have describe above, there isn't a hard rule to determine which case to choose.
I would have handled your exercise a bit differently. Instead of including all variables and removing one after the other, I would have instead included only the best variable first (the one with the highest absolute correlation with the dependent variable). Next, I would have looked at which of the remaining variables have the highest correlation with the residual of the model with the best variable. That is how to select your second best variable. And, you continue this process as long as your Adjusted R Square keeps on rising.
To go back to your question, which ones of the two models would I choose... I would go with the one with the highest adjusted R Square. That's because a variable does not have to be statistically significant to contribute incremental information to the model. That's as long as the direction or the sign of the coefficient makes sense. Here I disagree with other responders. I think if your regression coefficient is of opposite sign than the original correlation (contradicting economic theory or the logic underlying the model) you are heading for trouble. You will likely have an "overfit" model that will perform very poorly in forecasting.
Also, Goodness-of-fit is not the only criteria to select a model. And, it is an incomplete one at that. You also have to check the performance of your models in a Hold Out sample. You also should conduct sensitivity analysis. I mean by that stress the values of the independent variables associated with possible economic scenarios somewhat outside the norm but that are still possible. And, the best model is often the one that performs best in the Hold Out and through the sensitivity analysis testing. Meanwhile, this same model may have the weaker Goodness-of-fit (lower R Square). This happens all the time. A model with a lower R Square and fewer variables very often beats out one with more variables and a higher R Square... because the latter was simply overfit.
To go back to your original question, you can't select between your two models yet until you test both of them through a Hold Out sample and conduct some sensitivity analysis. Once you do that, a clear winner will readily emerge.