I'm a bit confused about the last step in a machine learning assignment. My task is to identify a model which best predicts a response variable.

I've got some data, and initial inspection shows that at least one of the independent variables is linearly related to the response variable. Great!

So I've run some model selection techniques (i.e. best subset, forward & backward stepwise regression, ridge and lasso regression) and I've identified the optimal number of variables to include the model (using cross validation), and the particular variables themselves.

Now do I insert these variables identified back into the multiple linear regression model to estimate the coefficients? And should this be on all the data or just a subset of the data? Or do I just use the regression coefficients estimated from the best subset regression (the method which had the lowest test MSE) ? The reason I'm hesitant to do the latter is I understand that best subset is a variable selection method and is not used for predictions -- is this correct?

  • $\begingroup$ I would think a self-study tag is appropriate there since this is part of an assignment. $\endgroup$
    – Xi'an
    Apr 7 '18 at 16:31

You split your data into two parts. FIRST PART is used for BOTH: variable selection AND estimating the coefficients.

SECOND PART: Do not touch.

So, run variable subset selection method on YOUR FIRST PART (50% - 80% of the data). This will give you the best predictors/variables. Lets assume here you're splitting it 50/50. There are 1000 rows.

Now, once you have the best variables say (v1, v2, v3, v4, v7) out of 100 variables, REMOVE all the other 95 variables from both your FIRST PART and SECOND PART. In implementation, you would chuck 95 columns and only keep columns which represent (v1, v2, v3, v4, v7, Y).

So now your FIRST HALF has (1 - 500 rows of):

(v1, v2, v3, v4, v7, dependentVariable ($Y_{train}$))

Now, estimate the coefficients using Normal Equations $W = (X^TX)^{-1}X^TY_{train}$, or gradient descent. Now using your estimates of the coefficients, ($W$) vector,

your SECOND HALF has (501 - 1000 rows):

(v1, v2, v3, v4, v7, dependentVariable ($Y_{test}$))

you now do

$Y_{pred} = W^T \times x$ (depends on your dimensions)

where $W$ is what you estimated using FIRST HALF, $x$ here represents the 501 - 1000 rows and column of predictors/variables in your SECOND HALF (v1, v2, v3, v4, v7). Now compare the $y_{pred}$, with the $Y_{test}$ column of SECOND HALF, that's your error.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.