My area of interest is Classification (Machine Learning). A Classifier will typically output a conditional probability distribution (P(Y=1|X) ) and the question may arise whether this predicted PDF corresponds to the actual observed frequencies of positive events. I.e. if a subset of 100 examples is rated by the Classifier with a probability to be positive in the neighborhood of 60%, are 60 of them indeed positive?
In this case, the Null Hypothesis is that the predicted PDF and the actual frequencies -- the empirical PDF-- differ. I.e. our predictions do not match reality. As is always the case the Null Hypothesis rejects the notion that we really know something.
Kolmogorov-Smirnov is a non-parametric test that allows us to test the Null Hypothesis that 2 distributions do not differ. Wikipedia has an interesting article on how this test should be conducted - a segment of which I copy below for the convenience of the reader:
My question is: How we should adapt this test when the Null Hypothesis is not that the 2 Distributions do not differ but exactly the opposite, that the 2 distributions are different?
Your advice will be appreciated.
#Replies to the comments made by Whuber and Dan Hicks
# #@Whuber:
#""As is always the case" is a misconception propagated by efforts at elementary explanation on the Web and in textbooks."
Could you please define precisely your disagreement and document it with citations of authoritative sources other than yourself?
"Furthermore, it's hard to see a hypothesis here at all"
Let's start from the basics: What is a Hypothesis in general and what is a Hypothesis in the Statistical Context?
According to the Merriam-Webster dictionary:
Hypothesis is an assumption or concession made for the sake of argument
According to the glossary of OECD (https://stats.oecd.org/glossary/detail.asp?ID=3675) :
Definition: A statistical hypothesis is a hypothesis concerning the parameters or from of the probability distribution for a designated population or populations, or, more generally, of a probabilistic mechanism which is supposed to generate the observations.
Source Publication: A Dictionary of Statistical Terms, 5th edition, prepared for the International Statistical Institute by F.H.C. Marriott. Published for the International Statistical Institute by Longman Scientific and Technical.
According to Wikipedia: (https://en.wikipedia.org/wiki/Statistical_hypothesis_testing) :
A statistical hypothesis, sometimes called confirmatory data analysis, is a hypothesis that is testable on the basis of observing a process that is modeled via a set of random variables.1 A statistical hypothesis test is a method of statistical inference. Commonly, two statistical data sets are compared, or a data set obtained by sampling is compared against a synthetic data set from an idealized model. A hypothesis is proposed for the statistical relationship between the two data sets, and this is compared as an alternative to an idealized null hypothesis that proposes no relationship between two data sets. The comparison is deemed statistically significant if the relationship between the data sets would be an unlikely realization of the null hypothesis according to a threshold probability—the significance level. Hypothesis tests are used in determining what outcomes of a study would lead to a rejection of the null hypothesis for a pre-specified level of significance. The process of distinguishing between the null hypothesis and the alternative hypothesis is aided by identifying two conceptual types of errors (type 1 & type 2), and by specifying parametric limits on e.g. how much type 1 error will be permitted.
I will next show that my statement is legitimate -- i.e. in accordance with the established meaning of the word Hypothesis both in the general Sense and in the specific Sense it has been given by the Scientific Statistical Community -- and therefore can not be dismissed as non-sensical as you and later Dan Hicks do.
So the question is: Does the following statement makes sense?
"In this case, the Null Hypothesis is that the predicted PDF and the actual frequencies -- the empirical PDF-- differ."
To make crystal clear that it is a legitimate phrase and makes sense in both the General and the Statistical contexts -- although that should have been clear to you from the very beginning-- I will proceed as follows:
Consider:
1) A Set X of Labeled Examples {1,...,n} some of which are labeled "Positive" (1) and the rest being labeled "Negative" (0). Assume that X is a Sample derived from a General Population of unknown distributional parameters.
2) A Random Variable P(X) which is a probability of x being positive, predicted by a certain Classifier. P takes a value for each Element of the Set X.
3) The Dataset "A" defined as the Number of Predicted Positive Examples within each percentile interval of P(X). This set is derived by multiplying the number of Examples falling within each percentile interval of P(X) by the average value of P(X) in the percentile interval.
4) The Dataset "B" defined as the Number of Actual Positive Examples within each percentile interval of P(X). This set is derived by calculating --i.e. counting-- the number of Examples from the Set X that are assigned probabilities P(X) falling within the percentile interval.
In the setting I just defined it should be obvious that the following apply:
1) A "probabilistic mechanism" is present generating observations. Both Sets A and B are the outcomes of two processes --defined above-- which are probabilistic since they both depend on a Random Variable (P(X)). More generally, we clearly have here two "designated populations" -- the Sets A and B.
2) These "populations" are associated with numerical values --i.e. the elements of Sets A and B are actually numbers-- and as such they represent 2 Numerical Distributions.
3) Obviously, the Sets A and B have parameters describing the respective distributions of the numerical values of their elements.
4) Stating "Sets A and B represent the same distribution" or alternatively "Sets A and B do not represent the same distribution" are obviously Hypotheses in both the General and the Statistical Meaning of the term Hypothesis - as shown above.
5) Stating "Sets A and B do not represent the same distribution" and consider this being the statement that should be refuted by the Evidence beyond doubt is equivalent to my original phrase: "In this case, the Null Hypothesis is that the predicted PDF and the actual frequencies -- the empirical PDF-- differ."
#@ Dan Hicks
#You state:
So, to test the hypothesis that θ≠.5θ≠.5, you would have to calculate Pr(Tn=.6|θ≠.5)Pr(Tn=.6|θ≠.5). Which is generally going to be intractable, if not incoherent.
Do you imply that testing whether a coin is biased is intractable given Data -- i.e. experiments? Of course it is! E.g., we can always calculate the probability of occurrence of a pattern of successes and failures under the assumption of a fair coin and then decide if this pattern is too extreme to retain the Hypothesis that the coin is fair.
Moreover you state:
First, hypothesis testing isn't usually used in a machine learning context. The typical goal of machine learning is to make accurate predictions about future observations, whereas hypothesis testing is a method for drawing inferences about unobserved parameters
From the discussion above --and more generally from the study of the subject matter-- should be clear that Classification is making inferences --i.e. produces Estimates-- about the parameters of the conditional probability distribution P(Y=1|X).
