1
$\begingroup$

So, I have a binary classification problem. The classes are fairly balanced and I have a separate training set and a test set. No matter what I try, both classification accuracy and the f1 score are always hovering around 45-55%. This means that the model is just random guessing and not learning anything. I had this doubt that this could be because these two classes are so similar i.e they are inseparable.

Therefore, I ran a Kolmogorov–Smirnov test and got a p-value of 0.87 between two classes for the training set and a staggering 0.98 for the test set. I have 20 independent variables and I know that K-S works only for 1-D data. Therefore, I just flattened the n-d array into a 1-D array so that I can perform a K-S on it.

My question is this, can I now conclude that these two classes are statistically so similar and are inseparable using machine learning algorithms ?. Is this claim so naive? Is Kolmogorov–Smirnov test enough of a proof to prove this?

Can two classes be truly inseparable even in a higher dimension? i.e via kernel transformation or neural nets.

I did use different regularization techniques as well just to make sure that this is not due to overfitting.

Improve this question
$\endgroup$
8
  • $\begingroup$ What are your K-S tests actually testing? $\endgroup$
    – jbowman
    Commented Nov 2, 2018 at 18:55
  • $\begingroup$ The similarly between two classes $\endgroup$
    – Ambarish
    Commented Nov 2, 2018 at 18:56
  • $\begingroup$ As @jbowman, I don't understand how you use K-S to assess the similarity between the two classes. Do you have only a single predictor, and do you run K-S on the distributions of this single predictor in the two classes? Also: Why is accuracy not the best measure for assessing classification models? and Is accuracy an improper scoring rule in a binary classification setting? $\endgroup$
    – Stephan Kolassa
    Commented Nov 2, 2018 at 21:53
  • 1
    $\begingroup$ Don't we use flattening in CNN? If this is not the right way then can you suggest me a way to compare two multivariate datasets? I need to check if they are similar or not. $\endgroup$
    – Ambarish
    Commented Nov 4, 2018 at 18:08
  • 1
    $\begingroup$ CNN is not performing a formal statistical test of a hypothesis - a very different thing than just fitting a model. $\endgroup$
    – jbowman
    Commented Nov 7, 2018 at 2:19

1 Answer 1

Reset to default
1
$\begingroup$

Your final question is "Can two classes be truly inseparable even in a higher dimension?"

It is certainly possible that two classes can't be separated by any algorithm. Two classes could, at least in theory, be absolutely identical, except for class. More likely, they could be almost identical.

Indeed, you want algorithms to not show difference where the differences are so small as to be trivial; you don't want to distinguish samples where the distinguishing characteristics are due to random chance. This is what p values are about. But, even if you have N = 10 billion and, therefore, any difference is statistically significant, you don't usually want to make distinctions where there is no appreciable difference.

For your earlier questions, I still don't see exactly what you are doing with the K-S tests. There are tests of equivalence that flip the usual roles of null and alternative hypotheses. One such is TOST - two one sided t tests. You will need to specify a difference that you consider so small as to be essentially equivalent.

I think when you did your original analysis and found "No matter what I try, both classification accuracy and the f1 score are always hovering around 45-55%" that you should have concluded that, indeed, none of the methods you were using worked. When nothing you are doing is working to (in this case) separate classes, then the two most common reasons are a) Your sample is too small or b) There just really isn't anything going on.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Peter for your clear explanation. As per me understanding K-S test is like any other hypothesis testing right? My idea was to use a no parametric test to check the equality of two distributions. In this case these are my two classes. Let's say I set the confidence level at 80 percentage, then if the p-value more than that I can accept my null hypothesis - two classes follow same distribution. I can't set the distance metric here because I don't know that. I can only use the information from p-value. $\endgroup$
    – Ambarish
    Commented Nov 4, 2018 at 12:32
  • $\begingroup$ But .... distribution of what? Usually the K-S is to test the difference in distribution of two continuous variables (or the same continuous variable on two groups). You haven't got that and I am not sure what you are getting when you flatten the data, or if that is suitable to K-S. My intuition is that it is not, but I don't know for sure. $\endgroup$
    – Peter Flom
    Commented Nov 4, 2018 at 12:36
  • $\begingroup$ Yes, exactly. Here the continuous variables are my independent variables. Flattering an array preserves it's information. So that I have the same continuous variable and two groups namely my two classes. $\endgroup$
    – Ambarish
    Commented Nov 4, 2018 at 12:42
  • $\begingroup$ OK, you may be right. It seems wrong, somehow, to me, but, as I said, I'm not sure. $\endgroup$
    – Peter Flom
    Commented Nov 4, 2018 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.