New answers tagged

0

I probably wouldn't call it "evidence", however I think it means "all the information in the data", which in Bayesian statistics is codified as $P(D)$ marginalised over all hypotheses/distributions deemed possible.


2

Conjugate priors may not be the best model choice if they are chosen for convenience (i.e., ease of deriving the posterior). Your priority -- ideally -- should be to select a prior that best describes your belief. For example, suppose you have a Poisson likelihood and have a prior belief that the rate parameter is uniformly likely to be (1,7), then you ...


1

Bayesian networks need to be acyclic. You don't put an arrow on every related node. Because, it is always the case that if $X$ is related to $Y$, $Y$ is related to $X$. Otherwise it'd be an undirected graph, or a directed graph with every relation doubled. Conceptually, causal relations make more sense when building the graph. And, the builder of the graph ...


1

This book hasn't been mentioned yet, as it's relatively new (2018): A Student's Guide to Bayesian Statistics by Ben Lambert In his videos, Lambert explains concepts in a very intuitive manner. This book does the same, explaining why the math works, using plenty of examples, and including problems to work at the end of each chapter. He incorporates videos ...


1

I think that the confusion stems from a somewhat ambiguous use of the terminology. In my view the term frequentism should be used for an interpretation of probabilities, namely that they refer to data generating processes in reality and correspond to limits of relative frequencies under idealised infinite repetition. In frequentism often parametric models ...


1

In my view, the author is noting that a Bayesian interprets probability $p(A)$ as quantification of our degree uncertainty about $A$, whether $A$ represents the outcome of a random event (e.g. result of a coin flipping) or e.g. the numerical value of a physical constant (e.g. the speed of light). Both things are though and represented using a single tool (...


1

The Metropolis-Hastings algorithm is applicable whatever the dimension of $\theta$. Finding the posterior of the joint distribution of $\theta_1$, $\theta_2$, $\theta_3$, and $\theta_4$, is equivalent to finding the posterior distribution of $\Theta$, where $\Theta$ is a $4 \times 1$ vector with $\Theta = [\theta_1 \ \theta_2 \ \theta_3 \ \theta_4]$. Let's ...


1

Yes, this is called the posterior predictive distribution. Mathematically, the histogram is approximating the following distribution $$ p(\tilde{y} \vert y) = \int p(\tilde{y} \vert \theta) p(\theta \vert y) \, d\theta$$ You'll note the first part of the integrand is the likelihood and the second is the posterior. This integral is integrating over all $\...


2

The quote does not say that if $\theta$ is a random variable, this is not Bayesian scenario. It says that the fact that $\theta$ is a random variable doesn't make it a Bayesian setting by itself. As already noticed in the comments, Bayes theorem is just a theorem in probability theory that lets us get the "reversed" conditional probability from $p(a|b)$ to $...


0

The property stems from the fact that a Dirichlet random variable $$X=(X_1,\cdots,X_k)\sim Dir_k(\alpha_1,\cdots,\alpha_k)$$ is distributed as$$X=(Y_1,\cdots,Y_k)/\sum_{i=1}^k Y_i$$ when $$Y_1,\cdots,Y_k\sim \prod_{i=1}^k \mathcal{Ga}(\alpha_i).$$ The result follows from the independence of the $Y_i$'s since $$\bigg(\frac{X_k}{\sum_{j\in\pi_1}X_j}\bigg)_{k\...


0

$$ \pi(\tau, \mu|\mathbf{x}) \propto \tau ^{\alpha + \frac{n}{2} - \frac{1}{2}}\text{exp}\Big(-\tau \Big[\beta + \frac{k}{2}(\mu - \nu)^2 + \frac{1}{2}\sum(x_i-\mu)^2\Big]\Big) $$ $$ \pi(\tau, \mu|\mathbf{x}) \propto \tau ^{\alpha + \frac{n}{2} - \frac{1}{2}}\text{exp}\Big(-\tau \Big[\beta + \frac{1}{2}(k + n)\Big(\mu-\frac{k\nu + n\bar{x}}{k + n}\Big)^2 +\...


0

Because Naive Bayes assumes your features are not correlated, you don't need to provide an explicit a priori causal model - or rather, you already have, just implicitly. A Naive Bayes algorithm for predicting cancer would assume $p(Cancer|Smoking,Tar) = p(Cancer|Smoking)p(Smoking) + p(Cancer|Tar)p(Tar)$ A Hierarchical Bayesian model could specify a model -...


1

hint $\int_0^{1} u^{a-1} (1-u)^{b-1} du=\beta(a,b)=\frac{\Gamma(a) \Gamma(b)}{\Gamma(a+b)}$ $$f(\theta |x)=\frac{\theta^{x} (1-\theta)^{n-x}}{\int_0^{1} u^{(x+1)-1} (1-u)^{(n-x+1)-1} du} =\frac{\theta^{(x+1)-1} (1-\theta)^{(n-x+1)-1}}{\beta(x+1,n-x+1)}$$ $$\theta \in (0,1)$$ For question 2) In the part $$\int_0^{1} u^{a-1} (1-u)^{b-1} du$$ It does not ...


1

One option would be to do survival analysis and use a Cox model to predict the probability that each individual already contacted will "survive" 30 days without responding, then subtract those predicted probabilities from 1 (with 0s for all the people who've already replied) and sum the results to get an estimate of the eventual response rate. In this model, ...


0

The point of using a surrogate model is that it lets you make an educated guess about what point to evaluate. Imagine that you are optimizing hyperparameters of a model that takes 24h to train, in such case, with large grid of hyperparameters, it could take years to train and evaluate the model for all the combinations of hyperparameters. So looking at the "...


1

The procedure, that you described, has a similar effect to artificially inflating the sample size. It's akin to cloning the observations of the sample. Say, you take each observation and create 9 more copies of it and run the Bayesian update on the 10 times bigger sample. Here's an example from the Wiki article. Suppose there is a school having 60% boys ...


3

As you note, Naive Bayes assumes the input features (predictors) are not correlated. This is a "naive" assumption, because input features commonly are correlated, just as regression predictors can be correlated (the problem of multicollinearity). But in some situations Naive Bayes models can work reasonably well and are much simpler to calculate.


1

In Bayesian inference we typically are interested in the conditional distribution of our parameters $\theta$ given data. In the case of regression, we condition on our predictors $X$ and outcomes $y$. Given known Gaussian noise with known variance, this reduces to inferring the conditional distribution of regression coefficients $\beta$ given the data $X,y$:...


2

I suggest reading this explanatory post on Bayes's experiment, but here's the main snippet: He [Bayes] used a thought experiment to illustrate the process. Imagine that Bayes has his back turned to a table, and he asks his assistant to drop a ball on the table. The table is such that the ball has just as much chance of landing at any one place on the ...


2

It's normal that you couldn't because it's not correct. Let $A=B'$. Then $P(A|B)=0$, which means $P(A|B)P(B|C)=0$, but the RHS is still $P(A|C)=P(B'|C)$, which is not necessarily $0$.


2

Perhaps GPML is no longer the best software for GPs, check out GPflow or GPy for Python implementations. I think the best way to do is to just override your hyp structure to have a likelihood variance of zero, remembering that it gets transformed by exp(), so use a large negative value. Then when you call gp function for predictive mean and you will get 0 ...


2

Your model is not identifaible. since $$f(y|\mu_1=1,\mu_2=2)=f(y|\mu_1=2,\mu_2=1)$$ but $$(1,2)\neq (2,1)$$ For resolving identifaibility problem, you should work with $\mu=\mu_1+\mu_2$ like $$y|\mu \sim N(\mu,\sigma^2)$$ or make a condition that guarantee identifaibility.


0

The rationale behind using a bayesian framework is not only the Bayes update rule or the availability of (subjective)prior if any exists, but it is due to the availability marginalization and conditioning, which drive the entire modeling process in Bayesian framework. These principles originate due to the quantification of uncertainty in the Bayesian ...


2

I'm not sure I really understand your analogy, but it seems like your analogy doesn't fully capture either the objections or the philosophy of Bayesianism. If I were going to criticize Bayesianism, I'd point to the fact that there are no guarantees on the results. Frequentists enjoy frequency properties of their estimators. They can say, at least in ...


0

The author compares $P(Y)$ (not $P(Y|X)$) and $P(X|Y)$ in terms of estimation complexity, due to the quickly growing $2^n$ term wrt number of dimensions. This is because of the dependence modelling in the joint distribution. Naive Bayes uses conditional independence assumption and relaxes this to $2n$ terms.


2

Not sure whether I understand precisely what you don't understand. My impression is that it just confuses you that one can speak about $p(x|\theta)$ as both a "proper pmf/pdf" (if interpreted as function over $x$) and a likelihood (if interpreted as function over $\theta$). The formula gives you the value for $p(\theta|x)$ for fixed values of $x$ and $\...


2

If integrated the conditional probability you would get $$ \int_\Theta p(\theta|x)d\theta = 1,$$ as expected - the posterior is a proper probability distribution, where I define proper to be that the integral over the parameter space is 1 and not just finite. But in many cases a probability distribution is in practice a product of bounded, positive ...


3

Usually, you set the tuning parameter once and evaluate the overall acceptance probability ex post once over all iterations of the sampler. Here, you aim for the ''golden acceptance ratio'' of 23,4%. In case your acceptance ratio is higher, this translates into the proposal variance being to small, leading to too many accepts as the posterior distribution is ...


0

I would like to wax philosophically on @kjetil's answer, and specifically this statement: That approximation will forget about the uncertainty in the estimation of θ, and might be a good approximation is some cases and bad in others. That must be evaluated on a case-by-case basis. The reason that we can use the MLE to good effect is because of the ...


1

Initially I wondered why you wanted to derive the full conditional for $\sigma$. But in the comments you say: My research adviser, in preparation for working with him, has given me the task to compare the a priori inverse-gamma with the log normal stated in the post. To this end, would it suffice to simply have the two analytical forms of the priors? ...


1

The model considered there is a mixture $$X\sim\sum_{i=1}^K \theta_i \sum_{j=1}^J \phi_{ij}\mathbb{I}_j(x)$$ The posterior distribution on the parameters of the model is thus $$\pi(\theta,\phi|x_1,\ldots,x_n)\propto\pi(\theta,\phi)\prod_{k=1}^n \sum_{i=1}^K \theta_i \sum_{j=1}^J \phi_{ij}\mathbb{I}_j(x_k)$$ with no manageable closed form expression for the ...


2

You seem to be talking about posterior predictive distribution, i.e. the a posteriori distribution of the data. You don't see $\theta$, because it is marginalized over possible parameter values. The distribution of the data given some particular parameter value is the likelihood function $P(x|i) = \phi_i(x)$. Regarding your comments, I guess the other thing ...


0

One particular problem you may run into is when your joint distribution is separated by regions of zero probability. Consider a bivariate distribution you are trying to sample from. Conditioning on one variable and sampling the other means you can only propose values in one dimension at a time. Consequently (as seen in the picture), your sampler will not be ...


1

For the simple model, $M_1$, the parameter is given ($0.5$), so you do not need to consider the prior probability -- the marginal likelihood is simply the given value. That is how we can justify the marginal likelihood for $M_1$. For the complex model, $M_2$, the likelihood is the Bernoulli distribution with respect to the prior, the given Beta distribution ...


0

I think, you are confused between the terms: pdf, pmf, probability Your understanding that, quote: "that PDFs are 0-valued at all individual points" is wrong. For a continuous random variable, probability it takes a certain value is 0, but pdf is not 0. The other minor thing: $p(\theta = 1 | Y = 16)$ is not a pdf, it is a pmf (conditional), because $\theta$...


2

The professor is not using a pdf, the professor is using a likelihood. Consider the following pdf: $$f(x_i|\mu)=\frac{1}{\pi}\frac{1}{1+(x_i-\mu)^2},\forall{x_i}\in\Re.$$ You are correct that $\Pr(x_1=5)=0$ as there is no area. However, the professor is not using a density, the professor is using a likelihood. Consider the following likelihood for the ...


1

From a theoretical perspective, the Bayesian comparison of $M$ models $\mathfrak M_m$ $m=1,...,M$ proceeds by the comparison of their posterior probabilities $$\pi(M_m|\mathbf x) \propto \pi(M_m) \int_{\Theta_m} f_m(\mathbf x|\theta) \pi_m(\theta_m)\,\text d\theta_m$$and therefore implies all priors over all models and over all model parameters. The ...


2

It's true any 3-variable probability distribution $p(A,B,C)$ can be factored as $p(A|B,C) p(B|C)p(C)$. If it is the case that $p(A| B, c) = p(A| B, c')$ for all $c,c' \in C$, then we have $p(A|B,C) = p(A|B)$ and $A$ is conditionally independent of $C$ given $B$. This conditional independence is represented in the graph structure by the lack of an edge from ...


3

The equation $p(A,B,C) = p(A\mid B,C)p(B\mid C)p(C)$ holds always. You don't need a Bayesian network or any other probabilistic structure. And $P(A|B,C)\neq P(A|B)$ in general. They're equal here just because of the Bayesian network's structure. So, you can't prove it. It's by definition. In a Bayesian network, a node is conditionally independent from any ...


0

In addition to Tim's answer, here are two general approaches to Bayesian hypothesis testing: you can also compute the highest density interval (HDI) or set of your posterior samples and see whether or not it contains 0 (if you're testing for $\theta=0$ for example). compute Bayes factor, which essentially gives you a ratio of the evidence of two competing ...


1

In Bayesian statistics we usually do not run classical null hypothesis testing, since we cal calculate things like $\Pr(\theta \gt 1 | X)$ directly. By using Bayes theorem, you calculate the posterior probability distribution for the parameter $$ p(\theta|X) \propto p(X|\theta) \, p(\theta) $$ and since you know the distribution, you can calculate the ...


2

Notice that the last three cases don't give exactly equal answers when properly rounded (e.g. in the last one, t-test gives 64.0% vs 64.1% in Bayes). If you have two iid. normally distributed groups of equal size, then the p-value is going to be equal to the Bayesian result, because they are the same function. The only difference lies in the interpretation. ...


4

Yes. A joint distribution $f_{X,Z}(x, z)$ of continuous variable $X \sim f_X$, and discrete variable $Z \sim p_Z$, is defined as any non-negative function of $x$ and $z$ that satisfies $$ \int f_{X,Z}(x, z) dx = p_Z(z), $$ $$ \sum_z f_{X,Z}(x, z) = f_X(x). $$ For a given distribution $f_{X,Z}$, the conditional distributions are defined: $$ p_{Z \mid X}(...


1

"denote this information X, and trivially assign probability 1.0 to Heads and 0.0 to Tails." You assign a probability to the frequency, the property how often the coin will flip heads/tails, and not to individual events that have occurred with the coin. Even when an event has already occurred we may assign a probability to it (the coin) that differs from ...


0

You seemed to have found the answer by yourself, so just for context, let's add some formality for you to understand it better. The distribution for coin flips is Bernoulli, so the likelihood is $$ f(x|p) = p^x \, (1-p)^{1-x} $$ where $x$ is tails encoded as $0$, or heads encoded as $1$, and $p \in [0, 1]$ is probability of observing heads. In frequentist ...


0

Later is better than never. Here is a natural and useful counterexample I believe, arising from Bayesian nonparametrics. Suppose ${\mathbf{x}} = \left( {{x_1},...,{x_i},...{x_n}} \right) \in {\mathbb{R}^n}$ has posterior probability distribution $p\left( {\left. {\mathbf{x}} \right|D} \right) \propto {e^{ - \frac{1}{2}{{\mathbf{x}}^{\mathbf{T}}}{\mathbf{Ax}...


0

Answering my own question :-) Re-reading this paragraph: P(A): The coin has a 50 percent chance of being Heads. P(A|X): You look at the coin, observe a Heads has landed, denote this information X, and trivially assign probability 1.0 to Heads and 0.0 to Tails. Given that I have already looked at the coin and know the answer (i.e. X), I would just ...


0

Yes and no. If your prior distribution is as you described, an improper flat prior, the posterior will take the shape of your likelihood; however, the likelihood still needs to be normalized. Conclusion: the posterior will look like the normalized likelihood. It is worth noting that flat priors are not invariant under transformation. That is you could ...


0

Bayes classifier chooses the class with maximum posterior probability. Since there are two classes, if $P(Y=1|X=x)\geq1/2$ it means posterior for class $1$ is the maximum one, and we choose it. That is why if $\eta(X)\geq 1/2$, it chooses class $1$. The formula for the posterior you've shared at the end is correct. You just need to be careful about calling ...


4

I believe it's because your priors are very different between the two scenarios. To see this, we first need to put the priors on the same interpretation (The first scenario talks about an individual, the the second scenario talks about a population). In the first scenario, your prior on a student knowing a correct answer is $P(X)=0.5$. If we extend this to ...


Top 50 recent answers are included