Using the mean of predict_proba outputs as an indicator of potential classifier accuracy for semi supervised learning Compare these code examples:
import lightgbm, numpy as np, pandas as pd
d = pd.DataFrame(np.random.randint(10, size=(10000,100)))
d['y'] = d.apply(lambda x: x[0] if np.random.random() < 0.55 else np.random.randint(10), axis = 1)
cl = lightgbm.LGBMClassifier().fit(d.iloc[:9000,0:10], d[:9000]['y'])
print(cl.score(d.iloc[9000:,0:10], d[9000:]['y']))
pr = cl.predict_proba(d.iloc[9000:,0:10])
print(np.mean([x.max() for x in pr]))



d = pd.DataFrame(np.random.randint(10, size=(10000,100)))
d['y'] = d.apply(lambda x: x[0] if np.random.random() < 0.95 else np.random.randint(10), axis = 1)
cl = lightgbm.LGBMClassifier().fit(d.iloc[:9000,0:10], d[:9000]['y'])
print(cl.score(d.iloc[9000:,0:10], d[9000:]['y']))
pr = cl.predict_proba(d.iloc[9000:,0:10])
print(np.mean([x.max() for x in pr]))

output
 0.59
 0.6095001350341738
 0.956
 0.9934019651423959

What I'm doing is training lgbm classifier with data that is very noisy, and taking the mean of the max predict_proba values.  I then do it again with data is that is only slightly noisy.
The mean of the predict proba max goes way up for the slightly noisy versus the very noisy data.
Is this a reasonable method of predicting the accuracy of a classifier without access to the labels?  I can't find any literature on this seemingly very important topic.  Papers, terminology, etc welcomed.
I understand this is a type of semi supervised learning, but can't find anything talking about this evaluation metric in particular.  I also understand that this might only roughly work under certain assumptions.  What would those assumptions be?
 A: Looking at your code, you have:

*

*Created a dataset where the label d['y'] (an integer from 0 to 9) is equal to the first column in the data, d[0], in $p$% of cases, and is a random value on the remainder.

*You've fit a classifier to the data, and shown that it's accurate $p$% of the time: it's right when the label matches the first column, and wrong when the label is set at random.

Unfortunately, all this tells you about the classifier is that it works well when one of the predictor columns is identical to the classification label, and badly when the classification label is picked at random. It might be a useful way of showing that an algorithm isn't working properly, but pretty much any properly-programmed classifier should behave in the same way here.
A: I may not be understanding your question correctly, so apologies if my answer is off topic.
Using predict_proba to construct a point prediction is acceptable.  However, to make a reliable probabilistic prediction (i.e. understand the accuracy of your prediction) you need to form a predictive p-value or construct a prediction interval. For example, a 75% prediction interval constructed from a sample of size $n$ will cover the next observed result 75% of the time in repeated sampling regardless of the unknown fixed true model parameters.
This type of probabilistic prediction is easiest to construct when the prediction target is a continuous endpoint.  When the sample space is discrete as in a logistic regression the possible error rates of the prediction interval will only come in discrete chunks.
