I am reading A note on the evidence and Bayesian Occam’s razor paper on model selection.
In page 3 it is mentioned that a prior $\mathcal{N}(0, 10^2)$ is used for each parameter $w_i$ of each model. The author goes on to mention that such a large variance makes our models put most of their mass on settings with sharp linear boundaries.
Can someone explain why is that?
How is the variance of our parameter's prior correlated to the form of our model's boundary?
As I read the paper, this is a classification problem. The choice of the prior was to be so adequately diffuse that it had only trivial impact on the parameters for each model. If the scaling of the problem were different, this might be a very tightly defined prior instead of a diffuse prior.
I think the choice of toy problems that they used was rather unfortunate. It made reading challenging unless you happen to work in this area, maybe. They created a Monte Carlo simulation of each of the $2^9$ possible plots and ordered them as per the heuristic in the appendix. They drew $10^8$ samples.
I think the observation you are making is an artifact of how they constructed the problem, the data, and the models. It is not intrinsically true outside this toy problem. They are just pointing out that a highly diffuse, but proper prior will result in almost no impact on the likelihood. In this case, as it is a classification problem, this acts like "edges."
If you look at the original McKay drawing in figure one, you can possibly see what they mean. Let us imagine that the true model is $H_1$ in the drawing. The mass falls off to nearly zero, effectively a boundary. Model $H_2$ being more diffuse would be disfavored. Conversely, had $H_2$ been the true model, but by chance, all of the data had been pulled from the far right of the model, Model $H_2$ would still have one since it does cover the data.
They point out that data set $h$ is not well modeled by any sharp linear boundary and so model $H_0$ (as they order them, not McKay) is the best model. If the data set highly resembled $h$ then the statement you are asking about is not true.
They were just trying to make sure the prior did not impact the choice of models or their parameters. It is quite possible that you could have a very diffuse model prior, but an unfortunate choice of priors on the parameters of some submodel, and get a bad prediction as a result. The danger in Bayes is that you will put either too little or too much information into the prior density. They chose a proper prior because you cannot know that the posterior would have been proper with flat priors. It probably would not have been the case, actually.
To give you an example of what they appear to mean, consider a classification problem where you needed to classify the font used in a document. This shouldn't be that difficult, just as constructing one to identify SKU's for a product should not be that difficult. These are operating under sharp well-defined rules.
Now imagine you needed to classify something very amorphous, such as the game people play when they see shapes in the clouds. It would be very difficult to settle on one pattern for a cloud. They didn't want the prior to choose the image in the cloud and that the choice of cloud image is clearest when it looks sharply like the object people think it looks like.
