Skip to main content
Cross Validated

Return to Question

Notice removed Draw attention by CommunityBot
Bounty Ended with no winning answer by CommunityBot
Notice added Draw attention by Roman
Bounty Started worth 50 reputation by Roman
LaTex and grammar
Source Link
Avraham
Avraham
  • 3.9k
  • 26
  • 46

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features (X)$X$ and a 1D array orof real-valued targets (Y)$Y$.

I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set of (X$X$ and Y)$Y$. As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features (X) and 1D array or real-valued targets (Y).

I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set (X and Y). As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features $X$ and a 1D array of real-valued targets $Y$.

I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set of $X$ and $Y$. As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.

Source Link
Roman
Roman
  • 724
  • 3
  • 28
  • 47

How to be sure that out-of-sample results of one model are really better than those of another model?

I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features (X) and 1D array or real-valued targets (Y).

I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set (X and Y). As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.