What is the problem of singular (non-invertible) covariance-variance matrix? What exactly is the problem of having non-invertible covariance matrix? Why is getting the inverse of this matrix so important? This problem is often encountered when doing regressiong works on samples, but even under context of sampling, I do not see how this becomes the problem..
 A: In short, if $\Sigma$, the covariance matrix in a multivariate normal distribution, is not invertible, then the density is not defined, as the multivariate normal density is 
$(2\pi)^{-k/2} |\Sigma| ^{-1/2} e^{(x - \mu)^T \Sigma^{-1}(x-\mu)/2 }$
Of course, this will not be defined if $\Sigma$ is not invertible. 
As @whuber points out, in different scenarios there are different heuristic reasons for why this would occur and why, heuristically, this leads to a problem in those scenarios. 
In terms of sampling, $sometimes$ this can be easily remedied. In particular $\Sigma$ will not invertible if the determinant = 0. In this case, at least one of the values of $X$ is a merely a linear combination of the other values. For the sake of example, let's suppose that $X_k$ = $\sum_{i = 1}^{k-1} X_i$, and the covariance of $X_{(-k)}$ (i.e. the $X_i$ s.t. $i < k$) is invertible. Then one can sample from $X_{(-k)}$ and then calculate $X_k$ from $X_{(-k)}$. So even though the density function is not properly defined, we can still draw samples from this distribution by being clever. 
A: Since you mark the regression tag, I assume that's the context. People are often interested in information like the following:
i) estimating a coefficient
ii) getting a standard error of an estimated coefficient
iii) testing an estimated coefficient
iv) getting a confidence interval or a prediction interval
these relate in some way to values in the inverse of the X'X matrix or equivalently, (after sweeping out the constant from X), the matrix of sums of squares and cross products which is a multiple of the variance covariance matrix of the predictors. You don't necessarily need to explicitly invert the matrix to get them, but if it's not invertible, they'll either be undefined, not uniquely defined, or infinite.
