1
$\begingroup$

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$.

I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set of $X$ and $Y$. As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Benavoli’s “Time for a Change” paper discusses classical methods for doing inference with cross-validation results. The paper then explains the advantages of Bayesian methods. You might, however, accept the drawbacks of the classical methods in exchange for ease of implementation and familiarity of techniques. // While the paper uses threshold-based metrics for classification problems, I do not recall any of the work being specific to classification accuracy, and I think a metric like $MSE$ could be compared with the methods that are discussed in the paper. $\endgroup$
    – Dave
    Commented Dec 27, 2021 at 14:37
  • $\begingroup$ there's no recipe for what you want, tough luck. the only way to ensure this happening is to get a dataset that has the same joint distribution that the model will be applied to in practice(production). but then there is no recipe for this to happen either $\endgroup$
    – Aksakal
    Commented Jan 6, 2022 at 18:57

1 Answer 1

Reset to default
1
$\begingroup$

Mostly people rely on Out of sample validation to be sure that their model performs better than a benchmark model. But to answer your questions you can maybe use the following hack:

  1. Step 1 : Run the model; let's say decision trees with 100 different initialisation keeping all other parameters as same. Once ran record the mean error on the out of sample data.

  2. Step 2 : Now you have results of 100 different models which should not vary that much as for most ML algorithms seeds just leads to slight variation.

  3. Step 3 : Perform a hypothesis testing on the model results as compared to benchmark model as mean (One Tailed Test) Null Hypothesis : Model results are similar to benchmark model

  4. Step 4 : If you get p-value less than 0.05 or 0.01 based on how strict you want to be. You can statistically reject the Null Hypothesis.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.