0
$\begingroup$

I am performing a regression classification to predict genes that are likely to cause disease. I have 600 rows of training genes by 8 features. Although only 50 genes have a score >0.9 (on a scale of 0 to 1, where 1 is definitely causing disease and 0 is a gene not causing the disease) and unfortunately cannot increase this, whilst I also have ~400 genes <0.1

I benchmark a few models using nested cross-validation and Bayesian hyperparameter tuning (models: gradient boosting, extreme gradient boosting, random forest, support vector machine, and k-nearest neighbors).

I assess model performance with all the metrics scikit-learn has to offer for regression classification (MSE, MAE, max error, r2, etc.). However when I re-run the same code I get small flucuations in these measures (e.g. r2 changes from 0.70 to 0.71.

Differences in performance have only been +/- 0.01 on my re-runs, but this concerns me as I thought I've done everything to make sure the performance is stable.

Is there anything I can do further check/ensure model performance stability? I thought about removing some of the genes that have <0.1 score to reduce the imbalance between them and the few genes that score >0.9 but I'm not sure if this is good practice? Also wouldn't go for synthetically increasing high scored genes as I'm not sure that's trustworthy either.

I have also:

  • Set the random_state seed globally (to 0)
  • I select the 8 features using the Boruta algorithm and check model performance further with shap
  • Checked that the imputation on the data before machine learning also outputs a stable dataset (so the data going into the models is the same every time). I use missforest random forest imputation for this.
  • For the tree-based models I don't scale or normalize data but for the other non-tree-based I do scaling.

Gradient boosting is my top model so in theory if its performance was stable I'd go for using that model, and so as an example the parameter tuning I do looks like:

seed=0
gbr = GradientBoostingRegressor(random_state=seed)
gbr_params = {
    'learning_rate': (0.01, 0.5),
    'max_depth': (1, 4),
    "max_features":["log2","sqrt", "auto"],
    "criterion": ["friedman_mse", "mse", "mae"],
    'n_estimators': (10, 50)
    }

inner_cv = KFold(n_splits=5, shuffle=True, random_state=seed)
outer_cv = KFold(n_splits=5, shuffle=True, random_state=seed)

I also use sklearn's train_test_split() for which I also set the same seed of 0.

$\endgroup$
0
$\begingroup$

Imho there is a global flaw, as you have not integrated for example imputing, e.g. with knn, and scaling within a pipeline. This may lead to overfitting, as the train data may see the test data e.g. from scaling, when doing it manually. From your code, I can not judge, what you are doing in detail. But you should base your predictions on a GridEstimator pipeline with SMOTE for inbalacing, scaling, and KNN: e.g. something like this:

#' split
X_train, X_test, y_train, y_test = train_test_split(X1, y, random_state = 42)

#Pipeline classifiers
classifiers = [RandomForestClassifier(), DecisionTreeClassifier()]

#' auto and manual selection'
steps_auto = [
    ('Impute KNN', KNNImputer(n_neighbors=3)), 
    ('SMOTE', SMOTE()), 
    ('RFTrees', classifiers[0])
] 


#' to add Feature Selection, if you want
steps_manual = [
    ('Impute KNN', KNNImputer(n_neighbors=3)), 
    ('SMOTE', SMOTE()), 
    ('Feature Selection', SelectFromModel(estimator=ExtraTreesClassifier())), 
    ('RFTrees', classifiers[0])
] 

pipeline_auto = Pipeline(steps_auto)
pipeline_manual = Pipeline(steps_manual)
#'choose params for grid/randomsearch'


params = {
          #'Trees__n_estimators': [100],
          'RFTrees__criterion': ["entropy"], #'entropy better'
          'RFTrees__max_depth': [3, 4, 5, 6, 7, 10, 15],
          'RFTrees__min_samples_split': [2, 3, 4, 10],
          'RFTrees__min_samples_leaf': [2, 3, 4, 10],
         }
     


grid_auto = GridSearchCV(pipeline_manual, param_grid=params, cv=10, n_jobs=-1, random_state = 42)
#' or

grid_manual = GridSearchCV(...)

I have not double checked the code, it is a template of myself, feel free to use it for your test and estimator or adapt it. And you need to add scaling, but pipelining makes everything relatively save in terms of subfolds and their values and sizes etc., in addition grid search really searches for the best solution...

If you really want to only care about accuracy you should check permutation importance of features to get a hint, which features you really need. If you want to have a look at the features individual contribution and how the data points affect your model: use the

shap library

After all that your model should be relatively stable, do not forget to tell your SMOTE which is the minority class

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.