# What does the prefit parameter in sklearn's SelectFromModel do?

I'd like to use sklearn's SelectFromModel to do feature selection. However, I'm not quite sure I understand the difference between prefit=True and prefit=False. According to the documentation:

Whether a prefit model is expected to be passed into the constructor directly or not.
If True, transform must be called directly and SelectFromModel cannot be used with cross_val_score, GridSearchCV and similar utilities that clone the estimator.
Otherwise train the model using fit and then transform to do feature selection.


I don't really understand what this means.

The prefit parameter of SelectFromModel indicates if the provided estimator has already been fit to data. If the parameter is set to True, the transform() method can be called directly to transform the data according to the most important features:

from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_classes=2, n_samples=100, n_features=5)

log_reg = LogisticRegression()
log_reg.fit(X, y)

selector = SelectFromModel(estimator=log_reg, prefit=True)

X_selected = selector.transform(X)


Alternatively, one can also choose to pass an estimator that has not been fitted to any data yet. In this case, SelectFromModel needs to first complete this step as well:

X, y = make_classification(n_classes=2, n_samples=100, n_features=5)

selector = SelectFromModel(estimator=LogisticRegression)

X_selected = selector.fit_transform(X)


The reason behind such a design is to make SelectFromModel compatible for use in data transformation pipelines, e.g. in cross-validation or grid search for hyperparameter tuning. Since in such pipelines a different subset of data points may be passed to the estimator in each iteration, it needs to be re-fit to each subset. This is also the reason why a SelectFromModel object with prefit=True cannot be used with e.g. cross_val_score or GridSearchCV.

Contrariwise, if SelectFromModel is not used in such pipelines and a fitted model is already available, setting prefit=True avoids re-fitting the estimator unnecessarily and saves computational resources.