1
$\begingroup$

I am not sure if I am following the steps correctly. Here, first I do a feature selection:

selection = SelectKBest(score_func=f_regression, k=15).fit(X,y)
X_features = selection.transform(X)

Then, I use cross-validation to calculate the alpha_ with the selected features (X_features):

model1 = LassoCV(cv=10, fit_intercept=True,  normalize=False, n_jobs=-1)
model1.fit(X_features, y)
myalpha = reg.alpha_

Then, I train a new model with the calculated alpha_ value:

model2 = linear_model.Lasso(alpha=myalpha)
model2.fit(X_features, y)

Finally, I use cross-validation to test the trained model on a new data set (test data):

pred_r = cross_val_predict(model2, X_test, y_test, cv=10)

I wonder if these steps are correct or not. For example, I am not sure if I need to do fit & transform on the test set as well. I appreciate any guidance.

UPDATE

I also wonder the scenario when there is no separate test set, and I need to do cross-validation on whole data:

selection = SelectKBest(score_func=f_regression, k=150).fit(X,y)
X_features = selection.transform(X)#

reg = linear_model.LassoCV(cv=10, fit_intercept=True,  normalize=False, n_jobs=-1)
reg.fit(X_features, y)

pred_r =  reg.predict(X_features)

With this code, I am afraid I am evaluating my model on the training set, which will result in biased results. Therefore, to decrease the bias and possibly overfit, instead of reg.predict(X_features), I need to do a cross validation again for testing:

pred_r =  cross_val_predict(reg, X_features, y, cv=10) 

I wonder if this would make sense?

$\endgroup$
2
$\begingroup$

When doing model selection, the order of steps should generally be:

  1. Decide which hyperparameters you want to cross-validate (based on prior knowledge, dataset size, computational budget...).
  2. Fix hyperparameters that are not cross-validated.
  3. Cross-validate remaining hyperparameters.
  4. Evaluate model on a test set.

In your case, the hyperparameters are a) the feature set, and b) alpha_ in the Lasso procedure. If you do not want to cross-validate the feature set (which, admittedly, is a hassle), then the order of steps you take is perfectly fine.

A few remarks / things to consider, unrelated to the order of steps:

  • The Lasso does implicit feature selection by setting some coefficients to zero, so maybe you can skip the SelectKBest step.
  • You do not have to refit the model, LassoCV does it automatically for you. So your model1 and model2 are identical.
  • If you have a separate test set, you do not need cross-validation for model evaluation. Just call model1.predict(X_test) and compare to y_test. You should preprocess the test data in exactly the same way that you preprocessed the training data (i.e. do not fit new transforms, only apply old ones):

    selection = SelectKBest(score_func=f_regression, k=15).fit(X,y)
    X_features = selection.transform(X)
    
    model1 = LassoCV(cv=10, fit_intercept=True,  normalize=False, n_jobs=-1)
    model1.fit(X_features, y)
    
    X_test = selection.transform(X_test)
    y_hat = model1.predict(X_test)
    square_loss = ((y_test - y_hat)**2).mean()  # compute your favourite metric
    
  • If there is no separate test set and you evaluate your model by CV, the one thing to bear in mind is that everything you do must be nested in a CV loop, including preprocessing, feature selection, etc. The pipeline structure makes this straightforward:

    from sklearn.pipeline import Pipeline
    
    pipe = Pipeline([  # enumerate all steps of the analysis here
        ('feature_sel', SelectKBest(score_func=f_regression, k=15)),
        ('lasso', LassoCV(cv=10, fit_intercept=True, normalize=False))
    ])
    
    yhat = cross_val_predict(pipe, X, y, cv=10)
    
| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I really appreciate your help Mary!!! Could you please have a look at the update in my question? I am not sure about my approach when there is no test data and when I need to do CV on whole data. Thank you again.. $\endgroup$ – renakre Mar 14 '17 at 15:36
  • $\begingroup$ Hi, @renakre ;) Your first suggestion indeed evaluates on the training data, and, as you say, it is preferable to do a global CV. When you do this, make sure that all steps of your analysis are nested inside the CV loop (as explained in the answer), otherwise the results will be biased. $\endgroup$ – mary Mar 15 '17 at 17:40
  • $\begingroup$ Thank you very much for updating your answer!!! I see that you used pipeline. So, in this case there is no need for calling the fit() or transform() methods $\endgroup$ – renakre Mar 15 '17 at 18:47
  • $\begingroup$ Exactly. cross_val_predict does so implicitly. $\endgroup$ – mary Mar 16 '17 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.