# Tag Info

## New answers tagged bayesian

4

It is called prior for a specific reason:

6

I intend to obtain the expectation and variance of my initial data and with them obtain shape1, shape2... That is wrong, and contradicts the methodology of Bayesian analysis. If you use your data to determine the hyperparameters in your prior distribution then you are double-counting the data --- using it once to get the prior distribution and then using ...

1

This is more a comment than an answer, but I hope it may be of some help. A difficulty, more frequent than one might think, in answering a probability & statistical question is that the question itself is actually ill-defined. And unfortunately our language doesn't help us in recognizing this problem: the question we ask is grammatically correct and seem ...

6

Maybe the confusion comes from the short hand $p(\theta|y)$ which actually means $p(\theta|Y=y)$, the random variable $Y$ interpreted as generating the data takes the fixed value $y$, fixed after actually having observed the data? So the data are random in the sense of having a distribution as long as they're uncertain, i.e., not fully observed, and then ...

0

To be concrete, consider the simple case of throwing a dice. Every face has a probability to be thrown. The outcome of all throws is non-random (it is a fixed pattern determined by throwing the dice a lot of times). On this pattern, you can apply a new chance of appearing. If you throw with two dices, a new pattern will emerge. This is because different ...

23

The Bayesian approach to (parametric) statistical inference starts from a statistical model, ie a family of parametrised distributions, $$X\sim F_\theta,\qquad\theta\in\Theta$$ and it introduces a supplementary probability distribution on the parameter $$\theta\sim\pi(\theta)$$ The posterior distribution on $\theta$ is thus defined as the conditional ...

0

This is late, but this theorem from 18.14 Asymptotic Statistics may help (not sure though): $\mathbf{Theorem:}$ A sequence of arbitrary maps $X_{n}: \Omega_{n} \mapsto \ell^{\infty}(T)$ converges weakly to a tight random element if and only if both of the following conditions hold. (i) The sequence $\left(X_{n, t_{1}}, \ldots, X_{n, t_{k}}\right)$ converges ...

1

Yes, that’s correct. Remember that the likelihood function varies the parameters, not the data. (See What is the difference between "likelihood" and "probability"?) The likelihood is a function of $\theta$ that assumes $\text{Data}$ is fixed, i.e. $$L(\theta \mid \text{Data}) = p(\text{Data} \mid \theta)\text{.}$$ What you have is now ...

1

The argument of "non-repeatability" is a red herring promoted by those on the extreme Bayesian side. After all, both frequentists and Bayesians start at precisely the same place: a model for potentially observable data given by $P(Y|X, \theta)$. This model states that the data we see is but one realization of a (potentially infinite) set of ...

3

The distribution for $\theta$ conditional on $s$ (and $n$) is \begin{equation} p(\theta|s) = \textsf{Beta}(\theta|s+1,n-s+1) . \end{equation} The problem is that we don't observe $s$. Instead we are given $\pi = (\pi_1, \ldots, \pi_n)$, where \begin{equation} p(x_i|\pi_i) = \textsf{Bernoulli}(x_i|\pi_i) \end{equation} and \begin{equation} p(x|\pi) = \...

4

Kernel density estimate is $$f(x) = \frac{1}{n} \sum_{i=1}^n \, K_h(x - x_i)$$ so to update it you don't need to do anything special, if a new $x_{n+1}$ point comes, just increment $n$ by one and append the array of $x_i$'s by a new value. We usually don't fit kernel density by memorizing and storing all the data, but we use tricks to approximate the ...

8

The problem does not seem to stand with MCMC but with the prior modelling. If the data comes from a Multinomial distribution $$\mathcal D_k(n,p_1,\ldots,p_k)$$ where the probability vector $\mathbf{p}=(p_1,\ldots,p_k)$ belongs to the $k$-dimensional simplex, the prior on $\mathbf{p}$ must put some mass on the simplex (and should logically be allocating all ...

1

That looks like the same thing $$\dfrac{0.3}{0.3\times0.001+0.995\times0.999}\times0.001$$ So yes you can write that. Or you could write $\Pr(\text{neg}\mid \text{inf})\dfrac{\Pr(\text{inf})}{\Pr(\text{neg})} = 0.3\times\dfrac{0.001}{0.3\times0.001+0.995\times0.999}$ If you want it to appear only once, you can also do with $$\Pr(\text{inf}\mid \text{neg})=\... 0 I would recommend you read West, Mike, and Jeff Harrison. Bayesian forecasting and dynamic models. Springer Science & Business Media, 2006 are there obvious theoretical issues with this setup? Yes, a few. A state space model is defined by an observation equation and a state equation. Terminology for these two equations varies across Kalman filter, ... 4 "But what if someone were to switch from a frequentist to a Bayesian viewpoint after constructing the CI and ask themselves the question: "How confident am I (i. e. at what rate would I - given that there is someone who knows μ and will reveal it at some point - be willing to bet) that this given CI contains μ knowing that it was constructed using ... 4 For so called location models, such as your linear regression, anovas etc., basically for models where the outcome depends linearly on the estimated parameters, the confidence interval will be the same as the credible interval with flat prior. If you want to know how would that credible interval look like with a different prior, then you add that prior ... -1 Can you maybe give some example how your "different datasets" look like? Are the predictors & the target variable the same in all datasets? Shot from the hip: Why don't you add the categorical features for "company", "region", etc. to your model and train only a single model on all the data? Besides, you should rather ... 1 Your scanned answer correctly derives the probability that a single test is positive. Now, you primarily need to consider how the probability of having two positive tests differs, and then you apply the same formula just with P(\text{two positive tests}|F_1) and P(\text{two positive tests}|F_2) instead of P(A|F_1) and P(A|F_2). Without additional ... 4 In general, they are not, even under the hypothesis that the transformation is monotonic. This is due to the fact that, when we transform variables, the density of the transformed variable is the determinant of the jacobian multiplied by the density of the original variable. The jacobian "reweights" the density in each point, so points that were in ... 6 Most MCMC algorithms do require the posterior density to be available up to a normalising constant, which means the likelihood \ell(\theta_t|x) to be available as well in the sense that it can be computed for the actual data x and an arbitrary value of the parameter \theta. For instance, the Metropolis-Hastings algorithm makes a proposal$$\theta'\sim q(...

1

Following suggestions from @Robert Long and @Henry (see the comments above), the issue is now solved. Apparently, reversing Boldness scores as I mentioned in my original question also reversed it's skew and so log transformation didn't help for Rboldness because it was negatively skewed. Therefore the model assumptions of linearity and heteroskedasticity ...

1

$\hat R$ doesn’t mean that your model converged. Is is one of several criteria that can be used to judge convergence. Usually you should not any of the criteria alone, but look at several different criteria. As noticed by Aki Vehtari et al (2019), there are cases where it fails. Moreover, when MCMC sampling you’re not only concerned about convergence but ...

0

As suggested by BruceET, you can turn your problem into an optimization. Let's consider that the beta parameters were $\alpha=2, \beta=4$, so $\mu = 1/3$ and the theoretical 95% CI is around $[0.052,0.716]$. Your problem is only having the statistics, and you want to find the parameters. mu <- 0.3333 ci <- c(0.052,0.716) We can define a cost function ...

5

If $X\sim\mathcal N(\mu,1)$ and $\mu\sim\mathcal N(0,10)$, the MAP of $\theta$ is $$\theta^\text{MAP}(x)=\dfrac{10}{11}x$$while the MAP of $\alpha=\alpha(\theta)=\exp\{\theta\}$ is $$\alpha^\text{MAP}(x)=\exp\left\{\dfrac{10}{11}x-\frac{\sqrt{10}}{\sqrt{11}}\right\}$$ The plot of the plugged-in density estimates $\varphi(\cdot;\theta^\text{MAP}(x),1)$ versus ...

3

Personally, I think at least some parameters seem suspicious, based on these traceplots and Rhat statistics. Especially the fourth plot on row one (with Rhat = 2.75) seems to show poor mixing. If I remember correctly, the Stan team indicates that any Rhat values above 1.01 could potentially be problematic. See, for instance, https://arxiv.org/abs/1903.08008 ...

6

"All models are wrong, but some are useful." - G.E.P. Box. If you wanted to have a realistic simulation of the universe, you would need to simulate each of its atoms, so you would need to have a computer larger than the universe itself. Every other simulation would be an approximation, making simplified assumptions. It would not be "correct&...

1

When describing the Gelman-Rubin statistic (R hat) in Bayesian Data Analysis (3rd Ed, ch. 11.4, pp. 283-285), Gelman and his coauthors say that the multiple chains used in calculating R hat should be simulated with overdispersed starting points and further that each chain should be split at the middle into two parts. For example, even if only two chains are ...

5

There is a typo¹ in the exponential part if not in the power: Starting from \begin{align} \pi((\mu, \sigma^2) | D) \propto ~ & \overbrace{\exp(-\sigma^{-2})(\sigma^2)^{-2}}^{\pi(\sigma^2)} \times \overbrace{\sigma^{-1} \exp(-\mu^2 / 2\sigma^2)}^{\pi(\mu|\sigma^2)} \times \\ &\underbrace{\exp(-(n(\mu - \bar{x})^2 + s^2)/2\sigma^2)/\sigma^n}_{\ell(\mu,\...

2

Basically, the Gelman-Rubin diagnostic looks at how different your repeated chains are. They are extremely similar in your case, giving you a relatively small GR diagnostic. However, at the same time your chains are obviously not converged and you need a longer burn-in (warm-up) period until you get actually useful posterior densities.

3

Detailed balance for a continuous Markov chain with transition kernel $K(\cdot,\cdot)$ and stationary density $f(\cdot)$ writes as $$f(x)K(x,y)=f(y)K(y,x)$$ It means that, in a stationary regime, the joint distribution of $(x_t,x_{t+1})$ is the same as the joint distribution of $(x_{t+1},x_{t})$. This implies that the chain $(x_t)$ is (time-)reversible.

6

I completely concur with Sycorax's comment that Adrian Raftery's 1988 Biometrika paper is the canon on this topic. How to derive analytically the negative log-likelihood (and its first-order conditions)? The likelihood is the same whether or not $n$ is unknown: $$L(n|y_1,\ldots,y_I)=\prod_{i=1}^I {n \choose y_i}p^{y_i}(1-p)^{n-y_i} \propto \dfrac{(n!)^I(1-... 0 To write P(A|B\cap C) in terms of P(B|A\cap C), you can use the definition of the conditional probability:$$P(A|B)=\frac{P(A\cap B)}{P(B)}$$Applying this defintion to P(A|B\cap C) and P(B|A\cap C) leads to the same term P(B \cap A\cap C) in the numerator, which you can thus substitute: \begin{eqnarray*} P(A|B\cap C) & = & \frac{P(B|A\cap ... 7 Several conceptual and R-coding errors: \mu_0 and \sigma_0 are hyper-parameters for the prior, hence should not be modified along iterations the conditionals in the Gibbs sampler are about the sampling model parameters \mu and \sigma which means updating mu=rnorm(... and sigma2=rinvgamma(... in the R code there was a factor 2 missing in n/2 in the ... 1 What you're proposing is to use time-varying effects or cofficients of covariates. To avoid confusion, restrict the phrase "time-varying covariates" to situations in which the values of covariates change over time. See the R time-varying vignette for the distinction. Before you proceed to time-varying coefficients, try to improve the model to ... 1 You are referring to the Metropolis algorithm. What it does, at each iteration it proposes a new value x' that is accepted with probability \min(p(x') / p(x), 1) what enables us to sample from the distribution p. This may look similar to simulated annealing, an optimization algorithm, but they are not the same. First, Metropolis generates a sample at ... 1 The more standard definition of Laplace smoothing is defined as adding a fixed quantity \alpha to each count$$ \hat p_i = \frac{n_i+\alpha}{\sum_{j=1}^K n_j+\alpha} = \frac{n_i+\alpha}{(\sum_{j=1}^K n_j) + K\alpha} $$where \alpha's can be thought as "pseudocounts", i.e. with \alpha=1 you assume that for each of the categories you observed ... 0 I will express your data as Y_{0:T} = (Y_0,\ldots,Y_T) where Y_0 is the initial value. I'm also going to assume there is a prior distribution for \lambda_0. (I'm guessing you made up a values for Y_0 and \lambda_0. That doesn't affect my suggested solution.) I'm not sure exactly how a sampler works for your model, but let's assume you have a valid ... 1 I have been developing the framework you are referring to. See https://github.com/audrey-b/simanalyse The R package has separate functions to simulate the datasets (using JAGS or R code), analyze (using JAGS) and evaluate the performance of the model. It can easily save all the results to files (thus saving on RAM) and parallelize on a local or cluster ... 0 is there any solid guideline for NHST Have you come across the special issue of the American Statistician with the ASA Statement on p-Values? 1 Literally, P(\text{data})=P(\mathcal D)=P(X_1=x_1...X_n=x_n) is the probability of obtaining the data you have. It is sometimes a probability density (e.g. when the data is continuous). In that case, we refer to it as f_{\mathbf X}(x_1...x_n) and the conditional density is referred as f_{\mathbf X|Y=y}(x_1...x_n). So, if you were to sample n data ... 1 P(data) That's how I read it: It's the probability of observing the given datapoint(s) in the population you are working on. For example, your dataset is a woman of age 75 (so data = 75) and in your study population there are 5% of women in the age bracket [74.5, 75.5) then P(data) = 0.05. If your dataset is two women of age 75 then P(data) = 0.05 * 0.05 = ... 2 But what does 𝑃(data|class=1) mean? Does it mean the probability of an age given the women has breast cancer? Yes, think about filter. Suppose you have a lot of data, each record has two fields, age and if this person had breast cancer. 𝑃(data|class=1) means we first filter the data with all people that had breast cancer, and then look the distribution of ... 2 The posterior on N is in the first case$$\pi(N|y)\propto \pi_N(N) \frac{N!}{(N-y)!} (1-p)^y\tag{1}$$and in the second case$$\pi(N|y_1,y_2) \propto \sum_{N_1=1}^{N-1} \frac{N_1!}{(N_1-y_1)!} \frac{(N-N_1)!}{(N-N_1-y_2)!} (1-p)^y\pi_{N_1,N_2}(N_1,N-N_1)\tag{2}$$Since$$\pi_N(N) = \sum_{N_1=1}^{N-1} \pi_{N_1,N_2}(N_1,N-N_1)$$there is no reason to believe ... 0 I stumbled across this post, when I tried to give a more comprehensive answer of that topic to a friend of mine. Since the answer written by @Tim♦ is already a bit older and this topic just really complex, I will try to give an easy introduction into the EM "algorithm" and its relationship to Maximum A Posteriori (MAP) and Maximum Likelihood ... 1 \log(\lambda) = \beta_0 for the male population. So you get a credible interval of \lambda by taking the exponentials of the bounds of the credible interval of \beta_0. For the female population, \log(\lambda) = \beta_0 + \beta_1. Adding the bounds of the credible intervals of \beta_0 and \beta_1 to get a credible interval of \beta_0 + \beta_1 ... 0 Please read my recent paper titled "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review". Visit link https://www.tandfonline.com/doi/full/10.1080/03610926.2021.1921214 Mean field assumption is NOT a modeling assumption. Rather, it is some imposition on top to a Bayesian model to implement coordinate ascent ... 1 This answer assumes that you are interested in computing \nabla_{\theta} \log Z rather than \nabla_{x} \log Z. Without foreclosing the possibility that there may exist other results relevant to your question, a key result that arises is when p(x) is in the exponential family. In this case, \log Z is referred to as the log-partition function or ... 1 You could take negatively-valued X and transform it into Y = -X, so that you could use a distribution with positive-only support for Y. You could any distribution with no such restriction, for example Gaussian, and truncate it from above at zero. There are many probability distributions and you can even come up with your own. It is an impossible task ... 4 This is not really an objection to Bayesian analysis at all --- it is an objection to the rules of probability theory. If you have an event \mathcal{E} (e.g., a head occurring on a coin) and a discrete parameter p giving the conditional probability of the event, then the law of total probability states that:$$\mathbb{P}(\mathcal{E}) = \sum \mathbb{P}(\...

6

What you refer to as $P(X=\text{head})$ is in fact an expected value $E(p)$ of a prior distribution that you've chosen for $p$. You noticed correctly that the distribution can take only two values $0.4$ and $0.6$, so $0.58$ is an impossible result for such distribution. There is nothing wrong with this. Bernoulli distribution can take only two values $0$ and ...

Top 50 recent answers are included