Best Subset Selection Questions I am reading Introduction to Statistical Learning and I have a question about the Best Subset Selection Algorithm. 
The algorithm goes like this: Suppose there are $p$ predictors, the aim is to select the best model that comes out using a subset of these predictors. 
First: Null Model. There is no predictor, just an intercept. 
Second: Out of $p$ possible predictors, fit all models containing only 2 predictors. This means there $p \choose 2$ ways. 
Third: Repeat the second step by increasing the number of chosen predictors along the way until you reach $p$. 

Here are my questions: 


*

*I am not sure what this 'model' results to. Do we literally regression equations from each of the predictors? For example, for the null model we have $Y=\beta_0$? And then for the second step, we get all possible combinations of $Y=\beta_0 + \beta_1X_1$, where $X_1$ is any of those predictors? 

*How do we determine the 'best' model from all these? If for example I have $p$ predictors and I have chosen 2 in the subset, I will have $p \choose 2$ sets of equations. How will I determine the 'best' model from these equations? 
Your insights are helpful. 
 A: Question 1
Yes, if we had three variables say (any more would be tedious to write out here) then all models containing 1 predictor would be fitted:
$$y_i = \beta_0 + \beta_1 x_{i1} + \varepsilon_i$$
$$y_i = \beta_0 + \beta_1 x_{i2} + \varepsilon_i$$
$$y_i = \beta_0 + \beta_1 x_{i3} + \varepsilon_i$$
Then all combinations containing two predictors
$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_1 x_{i2} + \varepsilon_i$$
$$y_i = \beta_0 + \beta_1 x_{i2} + \beta_1 x_{i3} + \varepsilon_i$$
$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_1 x_{i3} + \varepsilon_i$$
Then all combinations with three predictors, which in this case is just one, the full model
$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_1 x_{i2} + \beta_1 x_{i3} + \varepsilon_i$$
Question 2
How do you want to measure "best"? Commonly used measures include AIC, BIC, or Mallow's Cp, or the adjusted $R^2$ or plain old $R^2$. All of these will provide a ranking of the models in terms the features of that metric, but in practice they are all telling you about which model gives the best predictions (prediction accuracy or related) given the complexity of the model fit.
Having gone through this process, it would be difficult to then use the $p$ values of tests for terms in the model or an omnibus test for all terms as such tests would know nothing of the selection procedure you'd subjected the data to. It is also not clear how one might "correct" the p-values for the selection procedure with Best-subsets selection (Hastie et al 2009 - sorry I can't quite locate the page for this just now, will add if I find it later).
In addition, because you are using a hard threshold for inclusion (term is either in the model or not, in which case it's coefficient is $\beta_{ij} = 0$) the coefficients may well be biased. If the true coefficient is small, it may not get selected in the above process given the sample of data you have collected. In that case your model would say the estimate for the coefficient is 0, which is biased low compared to the true value. Variables that remain in the model could also have coefficients biased-high because the selection procedure is likely to retain those terms if they have large effects, and throw them out if they don't.
A: 1) Yes.
2) You evaluate by obtaining an estimate of the out of sample error rate for each of your models, and then choosing the model with the optimal estimate of out of sample error.  A few common methods for this are cross validation, using a fully held out data set, and some kind of estimate that can be performed on the training data itself that penalizes for the number of parameters.  In the first two cases, you have a model fit on training data, and a set of data you held out that the model has not seen.  You make predictions on the held out data, and then use a sample estimate of your loss function (often sum of squared residuals in the linear regression setting).  The third case subsumes methods like AIC and BIC, where a penalty term is applied to the training error rate.
All of these concepts are discussed in the book!  Keep reading!
