Testing when the Null Hypothesis is that the distributions do differ. Application in the case of Kolmogorov-Smirnov test My area of interest is Classification (Machine Learning).  A Classifier will typically output a conditional probability distribution (P(Y=1|X) ) and the question may arise whether this predicted PDF corresponds to the actual observed frequencies of positive events.  I.e. if a subset of 100 examples is rated by the Classifier with a probability to be positive in the neighborhood of 60%, are 60 of them indeed positive?
In this case, the Null Hypothesis is that the predicted PDF and the actual frequencies -- the empirical PDF-- differ. I.e. our predictions do not match reality. As is always the case the Null Hypothesis rejects the notion that we really know something.
Kolmogorov-Smirnov is a non-parametric test that allows us to test the Null Hypothesis that 2 distributions do not differ.  Wikipedia has an interesting article on how this test should be conducted - a segment of which I copy below for the convenience of the reader:

My question is:  How we should adapt this test when the Null Hypothesis is not that the 2 Distributions do not differ but exactly the opposite, that the 2 distributions are different?
Your advice will be appreciated.

#

Replies to the comments made by Whuber and Dan Hicks

#

#

@Whuber:

#


*

*
""As is always the case" is a misconception propagated by efforts at
  elementary explanation on the Web and in textbooks."

Could you please define precisely your disagreement and document it with citations of authoritative sources other than yourself?


*
"Furthermore, it's hard to see a hypothesis here at all"

Let's start from the basics:  What is a Hypothesis in general and what is a Hypothesis in the Statistical Context?
According to the Merriam-Webster dictionary:

Hypothesis is an assumption or concession made for the sake of
  argument

According to the glossary of OECD (https://stats.oecd.org/glossary/detail.asp?ID=3675)  :

Definition: A statistical hypothesis is a hypothesis concerning the
  parameters or from of the probability distribution for a designated
  population or populations, or, more generally, of a probabilistic
  mechanism which is supposed to generate the observations.
Source Publication: A Dictionary of Statistical Terms, 5th edition,
  prepared for the International Statistical Institute by F.H.C.
  Marriott. Published for the International Statistical Institute by
  Longman Scientific and Technical.

According to Wikipedia: 
 (https://en.wikipedia.org/wiki/Statistical_hypothesis_testing) :

A statistical hypothesis, sometimes called confirmatory data analysis,
  is a hypothesis that is testable on the basis of observing a process
  that is modeled via a set of random variables.1 A statistical
  hypothesis test is a method of statistical inference. Commonly, two
  statistical data sets are compared, or a data set obtained by sampling
  is compared against a synthetic data set from an idealized model. A
  hypothesis is proposed for the statistical relationship between the
  two data sets, and this is compared as an alternative to an idealized
  null hypothesis that proposes no relationship between two data sets.
  The comparison is deemed statistically significant if the relationship
  between the data sets would be an unlikely realization of the null
  hypothesis according to a threshold probability—the significance
  level. Hypothesis tests are used in determining what outcomes of a
  study would lead to a rejection of the null hypothesis for a
  pre-specified level of significance. The process of distinguishing
  between the null hypothesis and the alternative hypothesis is aided by
  identifying two conceptual types of errors (type 1 & type 2), and by
  specifying parametric limits on e.g. how much type 1 error will be
  permitted.

I will next show that my statement is legitimate -- i.e. in accordance with the established meaning of the word Hypothesis both in the general Sense and in the specific Sense it has been given by the Scientific Statistical Community -- and therefore can not be dismissed as non-sensical as you and later Dan Hicks do.  
So the question is: Does the following statement makes sense? 

"In this case, the Null Hypothesis is that the predicted PDF and the
  actual frequencies -- the empirical PDF-- differ."

To make crystal clear that it is a legitimate phrase and makes sense in both the General and the Statistical contexts -- although that should have been clear to you from the very beginning-- I will proceed as follows:
Consider:
1) A Set X of Labeled Examples {1,...,n} some of which are labeled "Positive" (1) and the rest being labeled "Negative" (0).  Assume that X is a Sample derived from a General Population of unknown distributional parameters.
2) A Random Variable P(X) which is a probability of x being positive, predicted by a certain Classifier.  P takes a value for each Element of the Set X.
3) The Dataset "A" defined as the Number of Predicted Positive Examples within each percentile interval of P(X).  This set is derived by multiplying the number of Examples falling within each percentile interval of P(X) by the average value of P(X) in the percentile interval.
4) The Dataset "B" defined as the Number of Actual Positive Examples within each percentile interval of P(X). This set is derived by calculating --i.e. counting-- the number of Examples from the Set X that are assigned probabilities P(X) falling within the percentile interval.
In the setting I just defined it should be obvious that the following apply:
1) A "probabilistic mechanism" is present generating observations.  Both Sets A and B are the outcomes of two processes --defined above-- which are probabilistic since they both depend on a Random Variable (P(X)).   More generally, we clearly have here two "designated populations" -- the Sets A and B.  
2) These "populations" are associated with numerical values --i.e. the elements of Sets A and B are actually numbers-- and as such they represent 2 Numerical Distributions.
3) Obviously, the Sets A and B have parameters describing the respective distributions of the numerical values of their elements.
4) Stating "Sets A and B represent the same distribution" or alternatively "Sets A and B do not represent the same distribution" are obviously Hypotheses in both the General and the Statistical Meaning of the term Hypothesis - as shown above.
5) Stating "Sets A and B do not represent the same distribution" and consider this being the statement that should be refuted by the Evidence beyond doubt is equivalent to my original phrase:  "In this case, the Null Hypothesis is that the predicted PDF and the actual frequencies -- the empirical PDF-- differ."

#

@ Dan Hicks

#

You state:

So, to test the hypothesis that θ≠.5θ≠.5, you would have to calculate
  Pr(Tn=.6|θ≠.5)Pr(Tn=.6|θ≠.5). Which is generally going to be
  intractable, if not incoherent.

Do you imply that testing whether a coin is biased is intractable given Data -- i.e. experiments?   Of course it is!  E.g., we can always calculate the probability of occurrence of a pattern of successes and failures under the assumption of a fair coin and then decide if this pattern is too extreme to retain the Hypothesis that the coin is fair.
Moreover you state:

First, hypothesis testing isn't usually used in a machine learning
  context. The typical goal of machine learning is to make accurate
  predictions about future observations, whereas hypothesis testing is a
  method for drawing inferences about unobserved parameters

From the discussion above --and more generally from the study of the subject matter-- should be clear that Classification is making inferences --i.e. produces Estimates-- about the parameters of the conditional probability distribution P(Y=1|X).    
 A: First, hypothesis testing isn't usually used in a machine learning context.  The typical goal of machine learning is to make accurate predictions about future observations, whereas hypothesis testing is a method for drawing inferences about unobserved parameters.  (See this answer.)  
Second, the p-value in a hypothesis test is the probability of observing data with the test statistic value $t = T$ given a hypothesis $H$:  $p = Pr(T = t | H)$.  When $H$ says that some parameter is equal to some particular value, this probability should be straightforward to calculate.  For example, say $\theta$ describes the probability that a coin comes up heads, $T_n$ is the fraction of coin flips that come up heads in $n$ trials, and in one particular experiment you observe $T_n = .6$, or 60% of the flips come up heads.  Then 
you can calculate $p = Pr(T_n = .6 | \theta = .5)$ to test the hypothesis that the coin is fair.  
So, to test the hypothesis that $\theta \neq .5$, you would have to calculate $Pr(T_n = .6 | \theta \neq .5)$.  Which is generally going to be intractable, if not incoherent.  
Following on whuber's comment, I think what you want is to estimate the accuracy of your classifier.  Read up on using cross-validation to do that.  
