# Tag Info

107

Well by distributing the $P(H)$ term, we obtain $$P(H|X) = \frac{P(X|H)P(H)}{P(X)} P(C) + P(H) [1 - P(C)],$$ which we can interpret as the Law of Total Probability applied to the event $C =$ "you are using Bayesian statistics correctly." So if you are using Bayesian statistics correctly, then you recover Bayes' law (the left fraction above) and if you aren'...

38

Similarity Fundamentally both types of algorithms were developed to answer one general question in machine learning applications: Given predictors (factors) $x_1, x_2, \ldots, x_p$ - how to incorporate the interactions between this factors in order to increase the performance? One way is to simply introduce new predictors: $x_{p+1} = x_1x_2, x_{p+2} = ... 31 Believe it or not, this type of model does pop up every now and then in very serious statistical models, especially when dealing with data fusion, i.e., trying to combine inference from multiple sensors trying to make inference on a single event. If a sensor malfunctions, it can greatly bias the inference made when trying to combine the signals from ... 18 Generally, informative priors are typically viewed as your information about parameters (or hypotheses) before seeing the data. So any data-based prior is violating the likelihood principle since evidence from the sample is coming through the likelihood function and the prior. 16 Well, if you are looking "for any pointers"... The (scaled)(inverse)Wishart distribution is often used because it is conjugate to the multivariate likelihood function and thus simplifies Gibbs sampling. In Stan, which uses Hamiltonian Monte Carlo sampling, there is no restriction for multivariate priors. The recommended approach is the separation strategy ... 12 I don't think this is an "overparamaterized" model at all. I would argue that by placing a prior over the Dirichlet paramaters, you're being less committal about any particular outcome. In particular, as you probably know, for symmetric dirichlet distributions (i.e.$\alpha_1 = \alpha_2 = ... \alpha_K$) setting$\alpha<1$gives more prior probability to ... 10 This is a term that is specifically from empirical Bayes (EB), in fact the concept that it refers to does not exist in true Bayesian inference. The original term was "borrowing strength", which was coined by John Tukey back in the 1960s and popularized further by Bradley Efron and Carl Morris in a series of statistical articles on Stein's paradox and ... 9 A similar problem is discussed in Gelman, Bayesian Data Analysis, (2nd ed, p. 128; 3rd edition p. 110). Gelman suggests a prior$p(a,b)\propto (a+b)^{-5/2}$, which effectively constrains the "prior sample size"$a+b$, and therefore the beta hyperprior is not likely to be highly informative on its own. (As the quantity$a+b$grows, the variance of the beta ... 9 The$p$-values are wrong. Take a simple example. Test whether a population mean$\mu$is equal to a particular value$\mu_0$or not. Suppose the sample mean$\bar x$is greater than$\mu_0$. Then it would be simply wrong to let the data guide you into testing only a one-sided alternative. Your$p$-value will be half of what it should be. And just to be ... 9 The expectation and variance computations in your example can be handled with the law of total expectation and law of total variance. The law of total expectation in your case reads: $$E(\theta) = E_{\mu} ( E_{\theta} (\theta \mid \mu ) )$$ where the subscripts indicate which variable is being averaged over in the expectation. The inside expectation is ... 9 Here are some relevant resources (full disclosure: the first link is to a paper of mine): http://newprairiepress.org/agstatconference/2014/proceedings/8 http://www.themattsimpson.com/2012/08/20/prior-distributions-for-covariance-matrices-the-scaled-inverse-wishart-prior/ http://andrewgelman.com/2012/08/29/more-on-scaled-inverse-wishart-and-prior-... 9 In my opinion, there are two different aspects to your question: when should I use a hierarchical model? when should I perform a Bayesian analysis? When should I use a hierarchical model? An advantage to using hierarchical models is their flexibility in modeling the continuum from all groups have the same parameters to all groups have completely different ... 8 NOT all the MCMC methods avoid the need for the normalising constant. However, many of them do (such as the Metropolis-Hastings algorithm), since the iteration process is based on the ratio$R(\theta_1,\theta_2)=\dfrac{\pi(\theta_1\vert x)}{\pi(\theta_2\vert x)}$, where $$\pi(\theta\vert x) = \dfrac{\pi(x\vert \theta)\pi(\theta)}{\int \pi(x\vert \theta)\pi(\... 8 Calculating posteriors with general/arbitrary priors directly may be a difficult task. On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given mixture of priors becomes the same mixture of the corresponding posteriors. [There are also many cases where some given prior may be quite well approximated ... 8 In my view, hierarchical modeling in a Bayesian setting mainly refers to the building of a complex prior structure. Consider a parameter of interest \theta_{0} and your observation (x_i). Now, consider for example that you are adding a supplemental layer to your model p(\theta_0|\theta_1) through hyperprior p(\theta_1) on \theta_1, then p(\... 8 The LKJ distribution is an extension of the work of H. Joe (1). Joe proposed a procedure to generate correlation matrices uniformly over the space of all positive definite correlation matrices. The contribution of (2) is that it extends Joe's work to show that there is a more efficient manner of generating such samples. The parameterization commonly used in ... 8 In plain English: The Beta distribution family is a set of continuous probability distributions. It describes random variables that can take values anywhere between 0 and 1. One example of a beta distribution is the uniform distribution on [0, 1]. A beta distribution has density proportional to x^{a-1}(1-x)^{b-1} where a and b are parameters. ... 8 TL;DR you can, but the result would strongly depend on your choice of prior. With maximum likelihood, you would be maximizing the likelihood, that in this case is defined in terms of probability mass function f of Bernoulli distribution, i.e. binomial distribution with number of trials n=1, parametrized by probability of success \theta$$ \hat\theta =... 7 Those two are my favorite questions about the subject: Weakly informative prior distributions for scale parameters What is an "uninformative prior"? Can we ever have one with truly no information? Also, it might help taking a look at section 2.9 of: Gelman, Carlin, Stern and Rubin (2004), Bayesian Data Analysis. And also Gelman's paper ... 7 I don't know for sure what the trick is, but this is my guess. Using JAGS syntax to specify$\xi \sim \mathcal D(\alpha)$, you would normally do something like this: xi ~ dirichlet(alpha[]) JAGS would then not allow you to assign a prior to$\alpha = (\alpha_1, \ldots, \alpha_J)$. Instead, let$\xi^\star_j \sim \mbox{Gamma}(\alpha_j, 1)$. Then it can be ... 7 Well, since you got your code to work, it looks like this answer is a bit too late. But I've already written the code, so... For what it's worth, this is the same* model fit with rstan. It is estimated in 11 seconds on my consumer laptop, achieving a higher effective sample size for our parameters of interest$(N, \theta)$in fewer iterations. raftery.... 7 In step 4, you don't have to reject the proposal$x,\theta$every time its new likelihood is lower; if you do so, you are doing a sort of optimization instead of sampling from the posterior distribution. Instead, if the proposal is worse then you still accept it with an acceptance probability$a$. With pure Gibbs sampling, the general strategy to sample ... 7 The hierarchical model you describe is a generative model. The model you constructed can be used to generate "fake" data. This is a little different conceptually than using your model to make predictions. The assumption underlying this concept is that a good model should generate fake data that is similar to the actual data set you used to make your model.... 7 First, note that I corrected the original wording of the question wrt the indicator functions in your likelihood definitions as they have to be functions of$x$not$\theta$. Hence the likelihood is $$f(x)=\theta x^{\theta-1}\mathbb{I}_{[0,1]}(x)$$ that clearly integrates to one: $$\int_0^1 \theta x^{\theta-1}\text{d}x = 1$$ Second, the posterior in$\theta$... 7 The theorem in question tells us that exchangeability is equivalent to being conditionally IID. Hence, in practice, data analysts consider the same things when deciding whether observations are exchangeable as when deciding whether they're (conditionally) independent. The basic approach is to treat as a covariate anything that might account for dependencies ... 7 Predictive here means predictive for observations. The prior distribution is a distribution for the parameters whereas the prior predictive distribution is a distribution for the observations. If$X$denotes the observations and we use the model (or likelihood)$p(x \mid \theta)$for$\theta \in \Theta$then a prior distribution is a distribution for$\...

7

The question has no clear answer because the empirical Bayes formulation does not & cannot specify how the hyperparameter is estimated. Take the simplest Normal mean estimation problem. When$$X\sim\mathcal N_p(\theta,I_p)\qquad\qquad\theta\sim\mathcal N_p(0,\sigma^2 I_p)$$the Bayes estimator of $\theta$ is\delta^\pi(x)=\frac{\sigma^2}{1+\sigma^2}x... 6 Ok, thanks to @Xi'an answer I could make the whole derivation. I will write it for a general case: \begin{align} \mathcal{W}(\mathbf{W} | \upsilon, \mathbf{S^{-1}} ) \times \mathcal{W}(\mathbf{S} | \upsilon_0, \mathbf{S_0}) \end{align} where the \mathbf{S^{-1}} is the key to conjugacy. If we want to use \mathbf{S} then it should be : \begin{align} \... 6 Your answer for the squared error loss part is wrong.\pi(\theta|x) \propto f(x|\theta) \pi(\theta) = 2\theta x^{\theta-1}I_{(0,1/2)}(\theta).  This is a $Beta(\theta,1)$ distribution in $x$, not in $\theta$, and the random variable in the posterior is $\theta$. So your answer is incorrect, and the correct answer would be the posterior mean of that ...

6

Important to note first: Bayesian inference does NOT automatically guard against overfitting. Adding additional variables will pretty much result in the same problems as in an non-Bayesian analysis. However, Bayesian model selection / model weights via marginal likelihood / Bayes factor CAN effectively include a penalty of model complexity, depending on how ...

Only top voted, non community-wiki answers of a minimum length are eligible