0
$\begingroup$

I have a point $x$ and a dataset $Y$ and I want to prove that $x$ cannot be getting from the dataset $Y$. More specifically, the point $x$ is a value I got from a classifier (it's not related to accuracy or probabilities, it interprets what the classifier has learned) and the data points in $Y$ were got by shuffling the labels and building the classifier model 1000 times. In other words, I'm trying to neglect the hypothesis that the point $x$ can be produced if you shuffled the labels, which points out that $x$ came out from the knowledge that the classifier has learned. I tried the t-test to prove that the $x$ is away from the mean of $Y$ but I'm not sure if this statistical test is sufficient.

$\endgroup$
1
  • $\begingroup$ No statistical test can prove a hypothesis; it can only provide information that might cause you to change your belief as to whether the hypothesis is indeed true. Those who already believe firmly in the truth of the hypothesis get nothing from the new knowledge; as far as they are concerned, the hypothesis is true. But for those who are more skeptical initially, the results of the experiment might cause them to be less skeptical (or it might not). $\endgroup$ – Dilip Sarwate Jul 4 '18 at 19:24
0
$\begingroup$

As was pointed out, you never prove the null-hypothesis. You reject it on the probability of the data arising given the null is true. This is the information the p-value gives you - it does not state the probability of the null-hypothesis given the data, it states the probability of the data under the null. This is an important distinction, as that is the fallacy of the transposed conditional.

It's not clear to me what the substantive hypothesis is regarding your algorithm. But if the probability of your classifier having learned from the training set is the probability of having selected a value lower / greater than returned,i.e., the complement, then you could estimate that with a z-score or t-score as you mentioned.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.