Drawing the Line between a Model and a parameters in Bayesian Model selection In Bayesian Model Selection, we first compute the evidence:
$$
p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta
$$
Then, we select the model that maximizes, which is a MAP estimator of the model.
$$
p(m|D) = \frac{p(D|m) p(m)}{\sum_m p(m,D)}
$$
If we assume the prior of the models $p(m)$ is uniform, this equates to selecting the model that maximizes $p(D|m)$. This is all fine.
But here is my question: how do we draw the line between parameters $\theta$ and model $m$. A more complex model can include simpler models. For example, a polynomial with 10 degrees of freedom clearly includes a polynomial with 3 degrees of freedom as we can just set the appropriate coefficients to 0 to get to the latter. Conversely, we can restrict parameters to a small set and treat each set as a model. At an extreme, we can say that every unique parameter $\theta$ value is a new model. In this sense,
$$
p(D|m) = p(D|\theta) 
$$
But here is the problem. We know that complex models tend to have larger likelihoods. So if we think of each unique parameter is a model, then the model that maximizes $p(D|m) = p(D|\theta)$ clearly overfits, defeating the purpose of model selection.
Of course, this is an extreme case to illustrate my point. But I feel that the line between model and parameters is blurred. We can have a lot of models with restrictive parameters, or fewer models with parameters taking wider ranges. And how we draw the line seems to affect the result of Bayesian model selection. In particular, if we tilt the line towards more models with restrictive parameters, it seems more prone to overfitting.
So in practice, how do people draw the line between models and parameters and go about Bayesian model selection?
A side question:
In both of the sources I cited below, evidence is computed as:
$$
p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta
$$
However, I wonder if the likelihood with regard to $\theta$ should be $p(D|\theta, m)$ instead of $p(D|\theta)$. I am not sure why it is considered not dependent on $m$.
References:

*

*http://alumni.media.mit.edu/~tpminka/statlearn/demo/

*MURPHY, KEVIN P. MACHINE LEARNING: a Probabilistic Perspective. MIT Press, 2012.

 A: In Bayesian model choice, the family of models under comparison is
$$\mathfrak F=\left\{p(\cdot|\theta_m,m)\,; m=1,\ldots,M,\,\theta_m\in\Theta_m \right\}$$
and the parameter of a member of this family is the pair $(m,\theta)$. I index $\theta_m$ by $m$ to stress that the parameter is model-dependent. Given a prior $\pi(\cdot)$ on the pair $(m,\theta)$, the posterior weight $p(m|D)$ is the marginal in $m$ of the joint posterior $\pi(m,\theta_m|D)$. The most likely model associated with $p(m|D)$ is a Bayesian decision procedure associated this posterior and a $0-1$ loss function, also known as MAP estimator, but only a Bayesian decision procedure rather than the outcome of a Bayesian analysis.
Concerning the worries about over-fitting, the evidence is responding to a larger parameter space by penalising a more poor fit thanks to the integration over this larger parameter space. This is connected with the intuition that distributions in larger dimensions are more concentrated, hence are more penalised when integrated out. The ratio of evidences known as Bayes factors are consistent tools for selecting a model and the first order approximation of the evidence known as BIC, Schwarz' criterion or the "Bayesian" information criterion, $\text{BIC}_m \approx −2\log p(D|m)$, is consistent as well.

This property of the (Bayesian) evidence to automatically penalise the more complex models is often called Ockham's razor, even though I personally dislike the term as non-quantitative. Some Bayesian papers promote a more balanced perspective, in comparison.
