what does one mean by numerical integration is too expensive? I am reading about Bayesian inference and I came across the phrase "numerical integration of the marginal likelihood is too expensive" 
I do not have a background in mathematics and I was wondering what exactly does expensive mean here? Is it just in terms of computation power or is there something more.  
 A: I will give you an example on discrete case to show why integration / sum over is very expensive.
Suppose we have $100$ binary random variables, and we have the joint distribution $P(X_1, X_2, \cdots, X_{100})$. (In fact, it is impossible to store the joint distribution in a table, because there are $2^{100}$ values. Let us assume we have the it in table and in RAM now.) 
To get a marginal distribution on $P(X_1)$, we need to sum over other random variables. (In continuous case, it is integrate over.)
$$P(X_1)=\sum_{X_2}\sum_{X_3}\cdots \sum_{X_{100}}P(X_1, X_2, \cdots, X_{100})$$
We are summing over $99$ variables, Therefore, there are exponentiation number of operations, in this case, it is $2^{99}$, which is a huge number that all the computers in earth will not able to do.
In probabilistic graphical models literature, such way of calculating marginal distribution is called "brute force" approach to perform "inference". By name, we may know it is expensive. And people use many other ways to perform the inference, e.g., getting the marginal distribution effectively. "Other ways" including approximate inference, etc.
A: Usually when performing Bayesian inference it's easy to encounter heavy integration  over nuisance variables for instance. Another example can be a numerical sampling as in this case from a likelihood function, meaning to perform a random sampling from a given distribution. As the number of model parameters increases, this sampling becomes extremely heavy and various computational methods have been developed to speed up the procedure and allow very fast implementations, keeping of course a high level of accuracy. These tecniques are for instance MC, MCMC, Metropolis ecc. Take a look in Bayesian data analysis by Gelman et. al it should give you a broad introduction! good luck
A: In the context of computational problems, including numerical methods for Bayesian inference, the phrase "too expensive" generally could refer to two issues


*

*a particular problem is too "large" to compute for a particular "budget"

*a general approach scales badly, i.e. has high computational complexity
For either case, the computational resources comprising the "budget" may consist of things like CPU cycles (time complexity), memory (space complexity), or communication bandwidth (within or between compute nodes). In the second instance, "too expensive" would mean intractable.
In the context of Bayesian computation, the quote is likely referring to issues with marginalization over a large number of variables.
For example, the abstract of this recent paper begins

Integration is affected by the curse of dimensionality and quickly becomes intractable as the dimensionality of the problem grows.

and goes on to say

We propose a randomized algorithm that ... can in turn be used, for instance, for marginal computation or model selection.

(For comparison, this recent book chapter discusses methods considered "not too expensive".)
