I need to solve a standard, let say "vanilla", regression problem: I have a 2D array of real-valued features (X) and 1D array or real-valued targets (Y).

I choose a simple model to fit (let say a linear regressor or a decision tree) and run a k-fold cross validation of a chosen model on my data set (X and Y). As a result I get an out-of-sample squared error that is smaller (better) than the squared error of a "constant model" (by "constant model" I mean a "model" in which always "predict" the same number independently on values of the features, which is just a mean of targets). How can I be sure that a better out-of-sample performance of my features dependent model is "real", and not just by chance? Maybe my model is statistically not better or even worse than the "constant model" and it gives better results just by chance.