We, bayesians, usually use non-informative priors for the parameters like
$p(\beta)\propto 1$.
Someone told me that such a flat prior is informative to some extent for non-linear functions of the parameters $\beta$. I thought this sort of prior was always uninformative for any kind of functional form.
I would appreciate some clarification on this point!
For any function there are many ways we can reparameterize it, for example a normal distribution $N(0, \sigma^2)$ can be described by its standard deviation or by its variance. If you use a uniform prior over the standard deviation you get a different posterior compared to using a uniform prior over the variance. If you square a uniform variable you don't get another uniform variable.
Therefore, although a uniform prior for the standard deviation appears to be uninformative we are adding information by choosing the standard deviation as the parameterization. Jeffrey's Prior is an attempt to solve this issue.
