Return to Answer

Let's see the (frequentist) maximum-likelihood point of view
From this point of view, speaking about the probability for a parameter value $x$ is simply meaningless. Full stop. We'd like to know what the true parameter value is, and the value $\tilde{x}$ that gives highest probability to the data $D$ should intuitively be not too far off the mark: $$\tilde{x} := \arg\max_x \mathrm{p}(D \mid x)\tag{*}\label{ML}.$$ This is the maximum-likelihood estimator.

Let's see the Bayesian point of view
From this point of view it always makes sense to speak of the probability for a continuous parameter, if we are uncertain about it, conditional on data and other evidence $D$. We write this as $$\mathrm{p}(x \mid D)\,\mathrm{d}x \propto \mathrm{p}(D \mid x)\, \mathrm{p}(x)\,\mathrm{d}x.\tag{**}\label{PD}$$ As remarked at the beginning, this probability refers to intervals on the parameter manifold, not to single points.

The estimator $\hat{P}$ associated with a utility function $G$ is the point that maximizes the expected utility given our data $D$. In a coordinate system $x$, its coordinate is $$\hat{x} := \arg\max_{x_0} \int G_x(x_0; x)\, \mathrm{p}(x \mid D)\,\mathrm{d}x.\tag{***}\label{UF}$$ This definition is independent of coordinate changes: in new coordinates $y=f(x)$ the coordinate of the estimator is $\hat{y}=f(\hat{x})$. This follows from the coordinate-independence of $G$ and of the integral.

1. the utility function above assumes a different expression, $G_y(y_0;y) = \delta[f^{-1}(y_0)-f^{-1}(y)] \equiv \delta(y_0-y)\,\lvert f'[f^{-1}(y_0)]\rvert$;
2. the prior density in the coordinate $y$ is not uniform, owing to the Jacobian determinant;
3. the estimator is not the maximum of the posterior density in the $y$ coordinate, because the Dirac delta has acquired an extra multiplicative factor;
4. the estimator is still given by the maximum of the likelihood in the new, $y$ coordinates.
   These changes combine so that the estimator point is still the same on the parameter manifold.

Let's see the (frequentist) maximum-likelihood point of view
From this point of view, speaking about the probability for a parameter value $x$ is simply meaningless. Full stop. We'd like to know what the true parameter value is, and the value $\tilde{x}$ that gives highest probability to the data $D$ should intuitively be not too far off the mark: $$\tilde{x} := \arg\max_x \mathrm{p}(D \mid x)\tag{1}\label{ML}.$$ This is the maximum-likelihood estimator.

Let's see the Bayesian point of view
From this point of view it always makes sense to speak of the probability for a continuous parameter, if we are uncertain about it, conditional on data and other evidence $D$. We write this as $$\mathrm{p}(x \mid D)\,\mathrm{d}x \propto \mathrm{p}(D \mid x)\, \mathrm{p}(x)\,\mathrm{d}x.\tag{2}\label{PD}$$ As remarked at the beginning, this probability refers to intervals on the parameter manifold, not to single points.

The estimator $\hat{P}$ associated with a utility function $G$ is the point that maximizes the expected utility given our data $D$. In a coordinate system $x$, its coordinate is $$\hat{x} := \arg\max_{x_0} \int G_x(x_0; x)\, \mathrm{p}(x \mid D)\,\mathrm{d}x.\tag{3}\label{UF}$$ This definition is independent of coordinate changes: in new coordinates $y=f(x)$ the coordinate of the estimator is $\hat{y}=f(\hat{x})$. This follows from the coordinate-independence of $G$ and of the integral.

1. the utility function above assumes a different expression: $G_y(y_0;y) = \delta[f^{-1}(y_0)-f^{-1}(y)] \equiv \delta(y_0-y)\,\lvert f'[f^{-1}(y_0)]\rvert$
2. the prior density in the coordinate $y$ is not uniform, owing to the Jacobian determinant;
3. the estimator is not the maximum of the posterior density in the $y$ coordinate, because the Dirac delta has acquired an extra multiplicative factor;
4. the estimator is still given by the maximum of the likelihood in the new, $y$ coordinates.
   These changes combine so that the estimator point is still the same on the parameter manifold.

[Update 29 March 2018: this answer has been largely rewritten after further research]

1. It can be useful to consider a parameter as a quantity living on a generic manifold, on which we can choose different coordinate systems or measurement units. From this point of view a reparameterization is just a change of coordinates. For example, the temperature of the triple point of water is the same whether we express it as $T=273.16$ (K), $t=0.01$ (°C), $\theta=32.01$ (°F), or $\eta=5.61$ (a logarithmic scale). Our inferences and decisions should be invariant with respect to coordinate changes. Some coordinate systems may be more natural than others, though, of course.

2. Probabilities for continuous quantities always refer to intervals (more precisely, sets) of values of such quantities, never to particular values; although in singular cases we can consider sets containing one value only, for example. The probability-density notation $p(x)\,\mathrm{d}x$, in Riemann-integral style, is telling us that
   (a) we have chosen a coordinate system $x$ on the parameter manifold,
   (b) this coordinate system allows us to speak of intervals of equal width,
   (c) the probability that the value lies in a small interval $\Delta x$ is approximately $p(x)\,\Delta x$, where $x$ is a point within the interval.
   (Alternatively we can speak of a base Lebesgue measure $\mathrm{d}x$ and intervals of equal measure, but the essence is the same.)

   Therefore, a statement like "$p(x_1) > p(x_2)$" does not mean that the probability at (or for) $x_1$ is larger than that at $x_2$, but that the probability of a small interval around $x_1$ is larger than the probability of an interval of equal width around $x_2$. Such statement is coordinate-dependent.

Let's see the (frequentist) maximum-likelihood point of view
From this point of view, speaking about the probability of a parameter value $x$ is simply meaningless. Full stop. We'd like to know what the true parameter value is, and the value $\tilde{x}$ that gives highest probability to the data $D$ should intuitively be not too far off the mark: $$\tilde{x} := \arg\max_x p(D \mid x)\tag{*}\label{ML}.$$ This is the maximum-likelihood estimator.

Let's see the Bayesian point of view
From this point of view it always makes sense to speak of the probability for a continuous parameter, if we are uncertain about it, conditional on data and other evidence $D$. We write this as $$p(x \mid D)\,\mathrm{d}x \propto p(D \mid x)\, p(x)\,\mathrm{d}x.\tag{**}\label{PD}$$ As remarked at the beginning, this probability refers to intervals on the parameter manifold, not to single points.

Ideally we should report our uncertainty by specifying the full probability distribution $p(x \mid D)\,\mathrm{d}x$ for the parameter. So the notion of estimator is secondary from a Bayesian perspective.

The estimator $\hat{P}$ associated with a utility function $G$ is the point that maximizes the expected utility given our data $D$. In a coordinate system $x$, its coordinate is $$\hat{x} := \arg\max_{x_0} \int G_x(x_0; x)\, p(x \mid D)\,\mathrm{d}x.\tag{***}\label{UF}$$ This definition is independent of coordinate changes: in new coordinates $y=f(x)$ the coordinate of the estimator is $\hat{y}=f(\hat{x})$. This follows from the coordinate-independence of $G$ and of the integral.

A utility function that in a particular coordinate system $x$ is equal to a Dirac delta, $G_x(x_0; x) = \delta(x_0-x)$, seems to do the job [3]. Equation $\eqref{UF}$ yields $\hat{x} = \arg\max_{x} p(x \mid D)$, and if the prior in $\eqref{PD}$ is uniform in the coordinate $x$, we obtain the maximum-likelihood estimate $\eqref{ML}$. Alternatively we can consider a sequence of utility functions with increasingly smaller support, e.g. $G_x(x_0; x) = 1$ if $\lvert x_0-x \rvert<\epsilon$ and $G_x(x_0; x) = 0$ elsewhere, for $\epsilon\to 0$ [4].

1. the utility function above assumes a different expression, $G_y(y_0;y) = \delta[f^{-1}(y_0)-f^{-1}(y)] \equiv \delta(y_0-y)\,\lvert f'[f^{-1}(y_0)]\rvert$;
2. the prior density in the coordinate $y$ is not uniform, owing to the Jacobian determinant;
3. the estimator is not the maximum of the posterior density in the $y$ coordinate, because the Dirac delta has acquired an extra multiplicative factor;
4. the estimator is still given by the maximum of the likelihood in the $y$ coordinates.
   These changes combine so that the estimator point is still the same on the parameter manifold.

Thus, the statement above is implicitly assuming a special coordinate system. A tentative, more explicit statement would could be this: "the maximum-likelihood estimator is numerically equal to the Bayesian estimator that has a delta utility function and uniform prior in some coordinate system".

In the Bayesian literature we can find also a more informal notion of estimator: it's a number that somehow "summarizes" a probability distribution, especially when it's inconvenient or impossible to specify its full density $p(x \mid D)\,\mathrm{d}x$; see e.g. Murphy [6] or MacKay [7]. This notion is usually detached from decision theory, and therefore may be coordinate-dependent or tacitly assumes a particular coordinate system. But in the a decision-theoretic definition of estimator, an number which is not invariant cannot be an estimator.

I apologize for the maybe excessive pedagogical tone; I hope it'll be useful if people who are learning these topics stumble upon this question.

1. It can be useful to consider a parameter as a quantity living on a generic manifold, on which we can choose different coordinate systems or measurement units. From this point of view a reparameterization is just a change of coordinates. For example, the temperature of the triple point of water is the same whether we express it as $T=273.16$ (K), $t=0.01$ (°C), $\theta=32.01$ (°F), or $\eta=5.61$ (a logarithmic scale). Our inferences and decisions should be invariant with respect to coordinate changes. Some coordinate systems may be more natural than others, though, of course.

2. Probabilities for continuous quantities always refer to intervals (more precisely, sets) of values of such quantities, never to particular values; although in singular cases we can consider sets containing one value only, for example. The probability-density notation $\mathrm{p}(x)\,\mathrm{d}x$, in Riemann-integral style, is telling us that
   (a) we have chosen a coordinate system $x$ on the parameter manifold,
   (b) this coordinate system allows us to speak of intervals of equal width,
   (c) the probability that the value lies in a small interval $\Delta x$ is approximately $\mathrm{p}(x)\,\Delta x$, where $x$ is a point within the interval.
   (Alternatively we can speak of a base Lebesgue measure $\mathrm{d}x$ and intervals of equal measure, but the essence is the same.)

   Therefore, a statement like "$\mathrm{p}(x_1) > \mathrm{p}(x_2)$" does not mean that the probability for $x_1$ is larger than that for $x_2$, but that the probability that $x$ lies in a small interval around $x_1$ is larger than the probability that it lies in an interval of equal width around $x_2$. Such statement is coordinate-dependent.

Let's see the (frequentist) maximum-likelihood point of view
From this point of view, speaking about the probability for a parameter value $x$ is simply meaningless. Full stop. We'd like to know what the true parameter value is, and the value $\tilde{x}$ that gives highest probability to the data $D$ should intuitively be not too far off the mark: $$\tilde{x} := \arg\max_x \mathrm{p}(D \mid x)\tag{*}\label{ML}.$$ This is the maximum-likelihood estimator.

Let's see the Bayesian point of view
From this point of view it always makes sense to speak of the probability for a continuous parameter, if we are uncertain about it, conditional on data and other evidence $D$. We write this as $$\mathrm{p}(x \mid D)\,\mathrm{d}x \propto \mathrm{p}(D \mid x)\, \mathrm{p}(x)\,\mathrm{d}x.\tag{**}\label{PD}$$ As remarked at the beginning, this probability refers to intervals on the parameter manifold, not to single points.

Ideally we should report our uncertainty by specifying the full probability distribution $\mathrm{p}(x \mid D)\,\mathrm{d}x$ for the parameter. So the notion of estimator is secondary from a Bayesian perspective.

The estimator $\hat{P}$ associated with a utility function $G$ is the point that maximizes the expected utility given our data $D$. In a coordinate system $x$, its coordinate is $$\hat{x} := \arg\max_{x_0} \int G_x(x_0; x)\, \mathrm{p}(x \mid D)\,\mathrm{d}x.\tag{***}\label{UF}$$ This definition is independent of coordinate changes: in new coordinates $y=f(x)$ the coordinate of the estimator is $\hat{y}=f(\hat{x})$. This follows from the coordinate-independence of $G$ and of the integral.

A utility function that in a particular coordinate system $x$ is equal to a Dirac delta, $G_x(x_0; x) = \delta(x_0-x)$, seems to do the job [3]. Equation $\eqref{UF}$ yields $\hat{x} = \arg\max_{x} \mathrm{p}(x \mid D)$, and if the prior in $\eqref{PD}$ is uniform in the coordinate $x$, we obtain the maximum-likelihood estimate $\eqref{ML}$. Alternatively we can consider a sequence of utility functions with increasingly smaller support, e.g. $G_x(x_0; x) = 1$ if $\lvert x_0-x \rvert<\epsilon$ and $G_x(x_0; x) = 0$ elsewhere, for $\epsilon\to 0$ [4].

1. the utility function above assumes a different expression, $G_y(y_0;y) = \delta[f^{-1}(y_0)-f^{-1}(y)] \equiv \delta(y_0-y)\,\lvert f'[f^{-1}(y_0)]\rvert$;
2. the prior density in the coordinate $y$ is not uniform, owing to the Jacobian determinant;
3. the estimator is not the maximum of the posterior density in the $y$ coordinate, because the Dirac delta has acquired an extra multiplicative factor;
4. the estimator is still given by the maximum of the likelihood in the new, $y$ coordinates.
   These changes combine so that the estimator point is still the same on the parameter manifold.

Thus, the statement above is implicitly assuming a special coordinate system. A tentative, more explicit statement would could be this: "the maximum-likelihood estimator is numerically equal to the Bayesian estimator that in some coordinate system has a delta utility function and a uniform prior".

In the Bayesian literature we can find also a more informal notion of estimator: it's a number that somehow "summarizes" a probability distribution, especially when it's inconvenient or impossible to specify its full density $\mathrm{p}(x \mid D)\,\mathrm{d}x$; see e.g. Murphy [6] or MacKay [7]. This notion is usually detached from decision theory, and therefore may be coordinate-dependent or tacitly assumes a particular coordinate system. But in the decision-theoretic definition of estimator, something which is not invariant cannot be an estimator.

Thus, the statement above is implicitly assuming a special coordinate system. When we change the latter the statement is not valid because the estimator is not the maximum of the posterior, and the prior is not uniform.

Final comments
The discussion above is sloppy, but can be made precise using measure theory and Stieltjes integration.

Thus, the statement above is implicitly assuming a special coordinate system. A tentative, more explicit statement would could be this: "the maximum-likelihood estimator is numerically equal to the Bayesian estimator that has a delta utility function and uniform prior in some coordinate system".

Final comments
The discussion above is informal, but can be made precise using measure theory and Stieltjes integration.