1
$\begingroup$

I am practicing fitting models to data sets that I could find on Kaggle but I don't know how these data sets were generated.

I remember there was a data set I played around with in order to fit a classifier. I tried lgbm because one of the categorical predictors has large cardinality and I did not want to do one-hot encoding. I also tried just fitting lgbm to numerical features only. I also tried catboost and simple linear regression.

After trying these algorithms, I noticed that the logloss on the test data was more or less the same for each algorithm and for each specification (deciding which features to include/exclude, etc.). I also found out that if for each sample in the test set, if I assign the predicted probability of being in class 0 and class 1 to be the proportion of samples in the data set belonging to each class, the logloss of this model without any predictors is roughly the same as the logloss I obtained with the machine learning algorithms taking into account various predictors.

This is telling me that the features may be insufficient to explain the output since I'm doing as well without them. At the same time, since I don't know how the data set was generated, I am unsure if the response was just randomly assigned. Is there any other way to verify this?

Improve this question
$\endgroup$
6
  • $\begingroup$ This is what significance tests are all about. $\endgroup$
    – whuber ♦
    Feb 21, 2022 at 15:58
  • $\begingroup$ I'm only familiar with significance tests for linear models. Is there one for other classifiers like decision trees? $\endgroup$
    – secondrate
    Feb 21, 2022 at 16:09
  • 2
    $\begingroup$ The concept is completely general: you can always test whether data were randomly assigned (this is called the "null hypothesis") either by mathematically working out the distribution of your results or, in more complex cases, by generating many independent such datasets, applying your analysis procedure to each one, and examining the distribution of results. $\endgroup$
    – whuber ♦
    Feb 21, 2022 at 16:13
  • $\begingroup$ I see. Do you have a reference you can point me to? A reference with an example would be very helpful. I'm not so familiar on this topic ... $\endgroup$
    – secondrate
    Feb 21, 2022 at 16:18
  • $\begingroup$ I think I can do a goodness of fit test on the response variable and check that it is Bernoulli with probability $p$ where $p$ is a constant value. If the features indeed mattered, $p$ wouldn't be constant, it would be a function of the features. Do I have the correct general idea? Thanks! $\endgroup$
    – secondrate
    Feb 21, 2022 at 17:15

1 Answer 1

Reset to default
0
$\begingroup$

Testing whether the response is unrelated to the predictors is the same as testing whether the distribution of the predictors is the same between response categories (i.e., they both test whether the response is independent of the predictors). A test that was specifically designed for this situation is the energy test for equal distributions, described in Rizzo & Székely (2016), which contains R code to implement the test. If you reject the null hypothesis of the test, there is evidence that the distribution of the predictors is associated with the response category in some way (though it doesn't tell you how; that is the job of the prediction model).

Improve this answer
$\endgroup$
1
  • $\begingroup$ This sounds interesting, but could you say a few words about how it might possibly work when one of the predictors has "large cardinality," as in the question? $\endgroup$
    – whuber ♦
    Feb 22, 2022 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.