Bayesian vs Frequentist: practical difference w.r.t. machine learning

I know that Bayesian and frequentist approaches differ in their definition of probability.

Practically, in machine learning a model is a formula with tunable parameters.

Then the difference between Bayesian and frequentist is:

That the parameters are assumed to be fixed numbers in frequentist setting and the parameters have their own distributions in the Bayesian setting.

Am I missing anything here or anything is mis-interpreted?

I am not asking theoretical arguments, just what is the practical manifestation of frequentist vs Bayesian w.r.t. Machine learning models and their optimization/fitting.

• Why do you say that they are different in their definition of probability ? Don't they use both the definition by Kolmogorov ? – user83346 Nov 19 '16 at 17:17
• @fcop i read belief vs frequency.... – Rafael Nov 19 '16 at 17:28
• Take a look at related threads in the column on the right. Maybe you will find an answer to your question there. – Richard Hardy Nov 19 '16 at 17:30
• Then is it 'definition' or 'interpretation' ? – user83346 Nov 19 '16 at 17:51
• Here you can read more about Bayesian way of looking at probability: stats.stackexchange.com/questions/173056/… – Tim Nov 19 '16 at 18:41

Once you've fitted the model, it will be what it will be, so I think the difference is prior to that. That is, the models / parameters are fitted differently between the Bayesian and Frequentist approaches. More specifically, the fitted Bayesian parameters will incorporate additional information outside of what is in the data. If you know something about what the parameters are likely to be (and you aren't wrong), that could boost the model's performance. Even if you use an 'uninformative' prior, you will typically find the fitted Bayesian parameters will be shrunk to some degree towards $0$ relative to the fitted Frequentist parameters.