How to identify outliers and drop rows in train splits of all folds, when using StratifiedKFold in GridSearchCV?

Question

For predicting whether a subject has liver disease or not, I'm using StratifiedKFold CV in GridSearch for AdaBoost and RandomForest Classsifiers.

For Outlier anlaysis, I've identified all feature outliers and extracted their row indices on full dataset but decided against dropping it since there might be some data leakage due to this strategy.

My next strategy was using log normalization to reduce the skewness, but the results are still sub-optimal. When maximized for precision, I'm onl touching the maximum of mean_average_score_ of 0.78 with AdaBost and 0.76 with RandomForest.

Need a new strategy for dropping outliers in 2 distinct sub-groups:

1. Has liver disease
2. Healthy

How do I perform outlier detection/drop only on the training split, of each Stratified fold of the data?

Kaggle Liver Patients Dataset

Answer 1

This is one approach I've come up with. Not sure how accurate this solution is.

import pandas as pd
import numpy as np
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report
from scipy.stats import zscore

# Load the dataset
df = pd.read_csv('indian_liver_patients.csv')  # Replace 'indian_liver_patients.csv' with the actual dataset file path

# Prepare the data
X = df.drop('Disease', axis=1)
y = df['Disease']

# Define the models and their respective parameter grids for GridSearchCV
adaboost = AdaBoostClassifier()
adaboost_params = {'n_estimators': [50, 100, 200], 'learning_rate': [0.1, 0.5, 1.0]}

random_forest = RandomForestClassifier()
random_forest_params = {'n_estimators': [50, 100, 200], 'max_depth': [None, 5, 10]}

# Perform GridSearchCV with StratifiedKFold cross-validation
cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)

# Define the outlier removal threshold
std_threshold = 2

# Custom scoring function to remove outliers within each fold
def custom_scoring(model, X, y):
    outlier_indices = set()
    for train_index, test_index in cv.split(X, y):
        X_train, y_train = X.iloc[train_index], y.iloc[train_index]
        X_test, y_test = X.iloc[test_index], y.iloc[test_index]

        # Iterate through continuous features
        for feature in X_train.select_dtypes(include=np.number).columns:
            # Compute z-scores for the feature within the training set
            z_scores = zscore(X_train[feature])

            # Identify outliers based on the z-scores and the threshold
            outliers = np.where(np.abs(z_scores) > std_threshold)[0]
            outlier_indices.update(X_train.iloc[outliers].index)

    # Remove outliers from the full dataset
    X_clean = X.drop(outlier_indices)
    y_clean = y.drop(outlier_indices)

    # Fit the model on the cleaned dataset and compute the score
    model.fit(X_clean, y_clean)
    y_pred = model.predict(X_test)
    score = accuracy_score(y_test, y_pred)

    return score

# GridSearchCV for AdaBoostClassifier
adaboost_grid = GridSearchCV(adaboost, adaboost_params, cv=cv, scoring=custom_scoring)
adaboost_grid.fit(X, y)

# GridSearchCV for RandomForestClassifier
random_forest_grid = GridSearchCV(random_forest, random_forest_params, cv=cv, scoring=custom_scoring)
random_forest_grid.fit(X, y)

# Print the best parameters for each model
print("Best parameters for AdaBoostClassifier:", adaboost_grid.best_params_)
print("Best parameters for RandomForestClassifier:", random_forest_grid.best_params_)

```