I have the polynomial regression model
$$Y_i | \mu, \sigma^2 \sim \mathcal{N}(\mu_i, \sigma^2), i = 1, \dots, n \ \text{independent}$$
$$\mu_i = \alpha + \beta_1 x_{i1} + \beta_2 x_{i2} + \beta_3 x^2_{i1} + \beta_4 x^2_{i2} + \beta_5 x_{i1} x_{i2}$$
$$\alpha \sim \text{some suitable prior}$$
$$\beta_1, \dots, \beta_5 \sim \text{some suitable prior}$$
$$\sigma^2 \sim \text{some suitable prior}$$
So far, I have chosen the uninformative priors
$$\alpha \sim \mathcal{N}(0, 1000^2)$$
$$\beta_i \sim \mathcal{N}(0, 1000^2)$$
based on previous examples of uninformative priors for regression models.
But how do I go about choosing a prior for $\sigma^2$? In other words, how does one make this determination?
I have come across examples such as $\sigma^2 \sim \text{half-Cauchy}(0, 5)$, but I am unsure what I should be using?
I would greatly appreciate it if people could please take the time to clarify this.
EDIT: In addition to Ben's response, I will add some useful information that I've found during my research.
All credit goes to Statistical Rethinking by Richard McElreath.
The Cauchy distribution is a useful thick-tailed probability distribution related to the Student t distribution. ... You can think of it as a weakly regularizing prior for standard deviations. ... But note that it is not necessary to use a half-Cauchy. The uniform prior will still work, and a simple exponential prior is also appropriate. In this example, as in many, there is so much data that the prior hardly matters.
...
Gelman (2006) recommends the half-Cauchy because it is approximately uniform in the tail and still weak near zero, without the odd behavior of traditional priors like the inverse-gamma. Polson and Scott (2012) examine this prior in more detail. Simpson et al. (2014) also note that the half-Cauchy prior has useful features, but recommend instead an exponential prior.
...
Gelman (2006) makes the following recommendations:
7 Recommendations
7.1 Prior distributions for variance parameters
In fitting hierarchical models, we recommend starting with a noninformative uniform prior density on standard deviation parameters $\sigma_\alpha$. We expect this will generally work well unless the number of groups $J$ is low (below 5, say). If $J$ is low, the uniform prior density tends to lead to high estimates of $\sigma_\alpha$, as discussed in Section 5.2. This miscalibration is an unavoidable consequence of the asymmetry in the parameter space, with variance parameters restricted to be positive. Similarly, there are no always-nonnegative classical unbiased estimators of $\sigma_\alpha$ or $\sigma_\alpha^2$ in the hierarchical model.
A user of a noninformative prior density might still like to use a proper distribution-- reasons could include Bayesian scruple, the desire to perform prior predictive checks (see Box, 1980, Gelman, Meng, and Stern, 1996, and Bayarri and Berger, 2000) or Bayes factors (see Kass and Raftery, 1995, O’Hagan, 1995, and Pauler, Wakefield, and Kass, 1999), or because computation is performed in Bugs, which requires proper distributions. For a noninformative but proper prior distribution, we recommend approximating the uniform density on $\sigma_\alpha$ by a uniform on a wide range (for example, $\text{U}(0, 100)$ in the SAT coaching example) or a half-normal centered at 0 with standard deviation set to a high value such as 100. The latter approach is particularly easy to program as a $\mathcal{N}(0,100^2)$ prior distribution for $\zeta$ in (2).
When more prior information is desired, for instance to restrict $\sigma_\alpha$ away from very large values, we recommend working within the half-$t$ family of prior distributions, which are more flexible and have better behavior near 0, compared to the inverse-gamma family. A reasonable starting point is the half-Cauchy family, with scale set to a value that is high but not off the scale; for example, 25 in the example in Section 5.2. When several variance parameters are present, we recommend a hierarchical model such as the half-Cauchy, with hyperparameter estimated from data.
We do not recommend the $\text{inverse-gamma}(\epsilon, \epsilon)$ family of noninformative prior distributions because, as discussed in Sections 4.3 and 5.1, in cases where $\sigma_\alpha$ is estimated to be near zero, the resulting inferences will be sensitive to $\epsilon$. The setting of near-zero variance parameters is important partly because this is where classical and Bayesian inferences for hierarchical models will differ the most (see Draper and Browne, 2005, and Section 3.4 of Gelman, 2005). Figure 1 illustrates the generally robust properties of the uniform prior density on $\sigma_\alpha$. Many Bayesians have preferred the inverse-gamma prior family, possibly because its conditional conjugacy suggested clean mathematical properties. However, by writing the hierarchical model in the form (2), we see conditional conjugacy in the wider class of half-$t$ distributions on $\sigma_\alpha$, which include the uniform and half-Cauchy densities on $\sigma_\alpha$ (as well as inverse-gamma on $\sigma_\alpha^2$ ) as special cases. From this perspective, the inverse-gamma family has nothing special to offer, and we prefer to work on the scale of the standard deviation parameter $\sigma_\alpha$, which is typically directly interpretable in the original model.
So it seems that Gelman (2006) is recommending, contrary to Ben's post, that we use either half-Cauchy, half-normal, or uniform priors instead of the inverse-gamma prior.
