Return to Question

Here is a complete code example using covtype dataset from sklearn (this is the first time I've tried this approach on this dataset):

import sklearn.datasets
cov = sklearn.datasets.fetch_covtype()

df = pd.DataFrame(cov.data)
df.columns = cov.feature_names
targ = 'Cover_Type'
df[targ] = cov.target
train = df.sample(frac=0.2).copy()
rest = df.drop(train.index).copy()
train_holdout = rest.sample(frac = 0.2).copy()
unlabeled = rest.drop(train_holdout.index).sample(frac=0.5)
for i in train, unlabeled, train_holdout:
    print(len(i))
len(train)+len(unlabeled)+len(train_holdout) == len(df)


import lightgbm, sklearn.model_selection, scipy.stats
X_holdout = train_holdout.drop(targ, axis = 1)
y_holdout = train_holdout[targ]
X = train.drop(targ, axis = 1)
y = train[targ]
X_test = unlabeled.drop(targ, axis = 1)
y_test = unlabeled[targ]
scs = []
for i in range(100, len(X), int(len(train)/20)):
    model = lightgbm.LGBMClassifier()
    #This is our model that we are evaluating
    model.fit(X[0:i],y.iloc[0:i])
    
    #Now we're going to distill knowledge from the unlabeled test set
    model_eval = lightgbm.LGBMClassifier()
    #We use the SSL pseudolabels to train with
    model_eval.fit(X_test,model.predict(X_test))
    #Loop multiple times on unlabeled test data for enhanced knowledge density
    for j in range(5):
        print(".", end="")
        model_eval1 = lightgbm.LGBMClassifier()
        model_eval1.fit(X_test,model_eval.predict(X_test))
        model_eval = lightgbm.LGBMClassifier()
        model_eval.fit(X_test,model_eval1.predict(X_test))
    #cv score for comparison
    cv_preds = sklearn.model_selection.cross_val_predict(lightgbm.LGBMClassifier(), X[0:i], y.iloc[0:i], cv=5, n_jobs=-1)
    roc = sklearn.metrics.accuracy_score(y.iloc[0:i], cv_preds)
    #This is our SSL metric
    v = sklearn.metrics.accuracy_score(y_holdout, model_eval.predict(X_holdout))
    #This is our ground truth score
    v2 = sklearn.metrics.accuracy_score(y_test, model.predict(X_test))
    print(i,v,v2, roc)
    scs.append([i, v,v2, roc])
    if len(scs) > 1:
        print("SSL Metric correlation", scipy.stats.pearsonr([x[1] for x in scs], [x[2] for x in scs]))
        print("CV correlation", scipy.stats.pearsonr([x[3] for x in scs], [x[2] for x in scs])) 
        print("CV + SSL correlation", scipy.stats.pearsonr([x[1]+x[3] for x in scs], [x[2] for x in scs])) 

df = pd.DataFrame(scs, columns = ["size", "SSL metric", "grnd score", "cv score"])
display(df.corr())
print("SSL Metric correlation", scipy.stats.pearsonr([x[1] for x in scs], [x[2] for x in scs]))
print("CV correlation", scipy.stats.pearsonr([x[3] for x in scs], [x[2] for x in scs]))





100 0.6066349691271702 0.6101848066952088 0.64
.....5910 0.7458746584625976 0.7838041350229126 0.7732656514382402
SSL Metric correlation (1.0, 1.0)
CV correlation (1.0, 1.0)
CV + SSL correlation (1.0, 1.0)
.....11720 0.7602138508207654 0.811412189927067 0.8033276450511946
SSL Metric correlation (0.9991068517535358, 0.0269084836575225)
CV correlation (0.9989002711826732, 0.029859170046122856)
CV + SSL correlation (0.9999959051170868, 0.0018218642469783922)
.....17530 0.7669800563671177 0.8208999376089154 0.8140901312036509
SSL Metric correlation (0.9989918045379589, 0.001008195462041117)
CV correlation (0.9983797333974359, 0.001620266602564091)
CV + SSL correlation (0.9999578462868633, 4.215371313665006e-05)
.....23340 0.7663776596889051 0.8284137604612637 0.8219794344473008
SSL Metric correlation (0.9976731160105244, 0.0001346927790430204)
CV correlation (0.998053283869585, 0.00010307680315341846)
CV + SSL correlation (0.9999315226935976, 6.802200040449242e-07)
.....29150 0.7730362943998623 0.8339751726511908 0.8266552315608919
SSL Metric correlation (0.997606186520421, 8.58865577639315e-06)
CV correlation (0.9979734083917876, 6.156448639323529e-06)
CV + SSL correlation (0.9999359217754913, 6.158896731084367e-09)
.....34960 0.77572556528474 0.8376110668875455 0.8301201372997712
SSL Metric correlation (0.9976059099140633, 5.379573298554177e-07)
CV correlation (0.9979292922627183, 3.743423127751737e-07)
CV + SSL correlation (0.9999368844933946, 6.078260889564408e-11)
.....40770 0.7685721047309654 0.839095544415998 0.8349766985528575
SSL Metric correlation (0.9956388999569111, 2.0668386326028505e-07)
CV correlation (0.9974544034623652, 4.1160363358897236e-08)
CV + SSL correlation (0.9998787077742654, 4.460662826642163e-12)
.....46580 0.7769518728082442 0.8406822142380758 0.8393945899527694
SSL Metric correlation (0.9957623303896034, 1.6243934882324038e-08)
CV correlation (0.9966095244747208, 7.447607149326285e-09)
CV + SSL correlation (0.9996882703419993, 1.7604708876086672e-12)
.....52390 0.7677760805490416 0.8430810438673867 0.8401221607176942
SSL Metric correlation (0.9930871025752668, 9.90857625137889e-09)
CV correlation (0.9962975371706202, 8.184836796820808e-10)
CV + SSL correlation (0.9995475274076026, 1.832784239150565e-13)
.....58200 0.7746068285966309 0.8350455024633722 0.8420274914089347
SSL Metric correlation (0.9931571058602614, 1.0513156735031128e-09)
CV correlation (0.9939105755200914, 6.226109128105513e-10)
CV + SSL correlation (0.9985711284072407, 9.204393460192126e-13)
.....64010 0.7840730621114004 0.838654503990878 0.8435713169817216
SSL Metric correlation (0.9920458296634339, 2.4743369407078367e-10)
CV correlation (0.9926253167154037, 1.6967659750598925e-10)
CV + SSL correlation (0.9971803735242705, 1.3969036471093728e-12)
.....69820 0.7679159226350551 0.8480830877132591 0.8407189916929246
SSL Metric correlation (0.9879449691152301, 2.907279904921447e-10)
CV correlation (0.9927929990874622, 1.7327452322624598e-11)
CV + SSL correlation (0.99621845913096, 5.024597051416664e-13)
.....75630 0.7756717798670425 0.8182214238075773 0.8436863678434484
SSL Metric correlation (0.9836951715303516, 2.6191890474789664e-10)
CV correlation (0.9833588368830564, 2.958397080779752e-10)
CV + SSL correlation (0.9892370594577319, 2.1930073622544316e-11)
.....81440 0.7792108603515415 0.8445171145199114 0.8405820235756385
SSL Metric correlation (0.9839456578957019, 4.102044996533527e-11)
CV correlation (0.9836510122857258, 4.613523813625284e-11)
CV + SSL correlation (0.9894169028452637, 2.7690159024078936e-12)
.....87250 0.7777801682407866 0.8465986101848066 0.845295128939828
SSL Metric correlation (0.9838842934546217, 7.255220825372512e-12)
CV correlation (0.9839113624449564, 7.1708579205882375e-12)
CV + SSL correlation (0.9895926542112893, 3.4500680328272935e-13)
.....93060 0.7733912781566662 0.8465502033088789 0.8463571889103804
SSL Metric correlation (0.9828849075303538, 1.964902980261665e-12)
CV correlation (0.9840779110411325, 1.1468137822713189e-12)
CV + SSL correlation (0.9896594578865701, 4.5767372484469556e-14)
.....98870 0.7725307114735053 0.8486908629332416 0.8469303125316071
SSL Metric correlation (0.9813064547790314, 7.071587344373537e-13)
CV correlation (0.9843172559170988, 1.7516885997717693e-13)
CV + SSL correlation (0.9895651454864534, 6.840699594696411e-15)
.....104680 0.7895914459671693 0.8482820937587401 0.8453859380970578
SSL Metric correlation (0.9805941828848543, 1.811828253020626e-13)
CV correlation (0.9845328429450951, 2.6697210024733667e-14)
CV + SSL correlation (0.9894789578513103, 1.0261775377329963e-15)
.....110490 0.7786837632581055 0.8485510208472279 0.844963345099104
SSL Metric correlation (0.9805666285652713, 3.4992180815384217e-14)
CV correlation (0.9847199739271252, 4.080332253477435e-15)
CV + SSL correlation (0.9895501835487054, 1.3587804611980958e-16)

I've been training on predicted labels and then using that to predict the train data, using the resulting score as a measure of fitness of my model.

It seems to work quite well in many different scenarios, though I do notice that some level of accuracy is required otherwise the score can be a bit noisy and less useful.

I'd like to learn more about this technique. Does it have a name or are they any references I can check out to better understand what I'm doing? I'd like to use it for epoch selection, forward feature selection, hill climbing, etc.

Note that this technique isn't really cross validation in the sense of oof / folding. It's more inverting the prediction process and a semi supervised learning coherence process.

Also note that I'm not proposing using predicted labels for training, just for measuring model fitness. I find that can lead to overfitting, though I suppose so can what I'm suggesting, at least indirectly.

example code:

model_to_be_evaluated = LGBMRegressor()
model_to_be_evaluated.fit(Xtrain, ytrain)
SSL_model_evaluator = LGBMRegressor()
SSL_model_evaluator.fit(Xtest,model_to_be_evaluated.predict(Xtest))
score_for_model_to_be_evaluated = SSL_model_evaluator.score(Xtrain, ytrain)

The idea being that I'm trying to blend in information regarding the unlabeled test data and how they interact with the predictions. I see significant correlation in many scenarios (not all) with CV score / test scores.

In some situations, all you have are predicted labels. The correlation is generally perfect against scores on ground truth at a coarse enough level in most scenarios, which makes it useful for some purposes.

At the very minimum, I find it very useful as a quick sanity check.

I've been training on predicted labels and then using that to predict the train data, using the resulting score as a measure of fitness of my model.

It seems to work quite well in many different scenarios, though I do notice that some level of accuracy is required otherwise the score can be a bit noisy and less useful.

I'd like to learn more about this technique. Does it have a name or are they any references I can check out to better understand what I'm doing? I'd like to use it for epoch selection, forward feature selection, hill climbing, etc.

Note that this technique isn't really cross validation in the sense of oof / folding. It's more inverting the prediction process and a semi supervised learning coherence process.

Also note that I'm not proposing using predicted labels for training, just for measuring model fitness. I find that can lead to overfitting, though I suppose so can what I'm suggesting, at least indirectly.

example code:

model_to_be_evaluated = LGBMRegressor()
model_to_be_evaluated.fit(Xtrain, ytrain)
SSL_model_evaluator = LGBMRegressor()
SSL_model_evaluator.fit(Xtest,model_to_be_evaluated.predict(Xtest))
score_for_model_to_be_evaluated = SSL_model_evaluator.score(Xtrain, ytrain)

The idea being that I'm trying to blend in information regarding the unlabeled test data and how they interact with the predictions. I see significant correlation in many scenarios (not all) with CV score / test scores.

In some situations, all you have are predicted labels. The correlation is generally perfect against scores on ground truth at a coarse enough level in most scenarios, which makes it useful for some purposes.

At the very minimum, I find it very useful as a quick sanity check.

edit to add that this might help with overfitting (Bjorn's point):

SSL_model_evaluator.score(Xtrain_holdout, ytrain_holdout)

I've been training on predicted labels and then using that to predict the train data, using the resulting score as a measure of fitness of my model.

It seems to work quite well in many different scenarios, though I do notice that some level of accuracy is required otherwise the score can be a bit noisy and less useful.

I'd like to learn more about this technique. Does it have a name or are they any references I can check out to better understand what I'm doing? I'd like to use it for epoch selection, forward feature selection, hill climbing, etc.

Note that this technique isn't really cross validation in the sense of oof / folding. It's more inverting the prediction process and a semi supervised learning coherence process.

Also note that I'm not proposing using predicted labels for training, just for measuring model fitness. I find that can lead to overfitting, though I suppose so can what I'm suggesting, at least indirectly.

example code:

model_to_be_evaluated = LGBMRegressor()
model_to_be_evaluated.fit(Xtrain, ytrain)
SSL_model_evaluator = LGBMRegressor()
SSL_model_evaluator.fit(Xtest,model_to_be_evaluated.predict(Xtest))
score_for_model_to_be_evaluated = SSL_model_evaluator.score(Xtrain, ytrain)

The idea being that I'm trying to blend in information regarding the test labels and how they interact with the predictions. I see significant correlation in many scenarios (not all) with CV score / test scores.

In some situations, all you have are predicted labels. The correlation is generally perfect against scores on ground truth at a coarse enough level in most scenarios, which makes it useful for some purposes.

I've been training on predicted labels and then using that to predict the train data, using the resulting score as a measure of fitness of my model.

It seems to work quite well in many different scenarios, though I do notice that some level of accuracy is required otherwise the score can be a bit noisy and less useful.

I'd like to learn more about this technique. Does it have a name or are they any references I can check out to better understand what I'm doing? I'd like to use it for epoch selection, forward feature selection, hill climbing, etc.

Note that this technique isn't really cross validation in the sense of oof / folding. It's more inverting the prediction process and a semi supervised learning coherence process.

Also note that I'm not proposing using predicted labels for training, just for measuring model fitness. I find that can lead to overfitting, though I suppose so can what I'm suggesting, at least indirectly.

example code:

model_to_be_evaluated = LGBMRegressor()
model_to_be_evaluated.fit(Xtrain, ytrain)
SSL_model_evaluator = LGBMRegressor()
SSL_model_evaluator.fit(Xtest,model_to_be_evaluated.predict(Xtest))
score_for_model_to_be_evaluated = SSL_model_evaluator.score(Xtrain, ytrain)

The idea being that I'm trying to blend in information regarding the unlabeled test data and how they interact with the predictions. I see significant correlation in many scenarios (not all) with CV score / test scores.

In some situations, all you have are predicted labels. The correlation is generally perfect against scores on ground truth at a coarse enough level in most scenarios, which makes it useful for some purposes.

At the very minimum, I find it very useful as a quick sanity check.