Quantum Mechanics has generalized probability theory to negative/imaginary numbers, mostly to explain interference patterns, wave/particle duality and generally weird things like that. It can be seen more abstractly, however, as a noncommutative generalisation of Bayesian probability (quote from Terrence Tao). I'm curious about these things, though by no means an expert. Does this have any applications outside Quantum Mechanics? Just curious.
Yes. I like the article Søren shared very much, and together with the references in that article I would recommend Muckenheim, W. et al. (1986). A Review of Extended Probabilities. Phys. Rep. 133 (6) 337-401. It's a physics paper for sure, but the applications there are not all related to quantum physics.
My personal favorite application relates to de Finetti's Theorem (also Bayesian in flavor): if we don't mind negative probabilities then it turns out that all exchangeable sequences (even finite, perhaps negatively correlated ones) are a (signed) mixture of IID sequences. Of course, this itself has applications in quantum mechanics, in particular, that Fermi-Dirac statistics yield the same type of (signed) mixture representation that Bose-Einstein statistics do.
My second personal favorite application (outside of physics proper) relates to infinite divisible (ID) distributions, which classically includes normal, gamma, poisson, ... the list continues. It isn't too hard to show that ID distributions must have unbounded support, which immediately kills distributions like the binomial or uniform (discrete+continuous) distributions. But if we permit negative probabilities then these problems disappear and the binomial, uniform (discrete+continuous), and a whole bunch of other distributions then become infinitely divisible - in this extended sense, please bear in mind. ID distributions relate to statistics in that they are limiting distributions in generalized central limit theorems.
Presumably there is a bunch of material on arXiv, too, though I haven't checked there in quite some time.
As a final remark, whuber is absolutely right that it isn't really legal to call anything a probability that doesn't lie in $[0,1]$, at the very least, not for the time being. Given that "negative probabilities" have been around for so long I don't see this changing in the near future, not without some kind of colossal breakthrough.
QM does not use negative or imaginary probabilities: if it did, they would no longer be probabilities!
What can be (and usually is) a complex value is the quantum mechanical wave function $\psi$. From it the probability amplitude (which is a bona fide probability density) can be constructed; it is variously written $<\psi|\psi>$ or $\|\psi\|^2$. When $\psi$ has (complex) scalar values, $\|\psi\|^2 = \psi^* \psi$. In every case these values are nonnegative real numbers.
For details, see the section on "Postulates of Quantum Mechanics" in the Wikipedia article.
I'm of the opinion that the "What's the application of this theory?" is a question that students of a theory should have to answer. Professor McGonagall spends all her time teaching and researching, it's up to her students to go find a use for the stuff in the world. (at least that's a kind-of defensible position, and the view I'll take just now)
So perhaps the question should be: first, understand the algebra of quantum interactions (von Neumann algebra); then, look for things in the world which behave this way. Instead of "Who else has already done this work?"
That said, one example that's tantalised me for a few years is V Danilov & A Lambert-Mogiliansky's use of von Neumann algebra in decision theory. Explicitly it is not about "quantum mechanics in the brain". Rather that "interfering (mental) states" might be a more accurate explanation of consumer behaviour than the usual picture: