50
$\begingroup$

I'm hoping that someone can explain, in layman's terms, what a characteristic function is and how it is used in practice. I've read that it is the Fourier transform of the pdf, so I guess I know what it is, but I still don't understand its purpose. If someone could provide an intuitive description of its purpose and perhaps an example of how it is typically used, that would be fantastic!

Just one last note: I have seen the Wikipedia page, but am apparently too dense to understand what is going on. What I'm looking for is an explanation that someone not immersed in the wonders of probability theory, say a computer scientist, could understand.

Improve this question
$\endgroup$

5 Answers 5

Reset to default
66
$\begingroup$

Back in the day, people used logarithm tables to multiply numbers faster. Why is this? Logarithms convert multiplication to addition, since $\log(ab) = \log(a) + \log(b)$. So in order to multiply two large numbers $a$ and $b$, you found their logarithms, added the logarithms, $z = \log(a) + \log(b)$, and then looked up $\exp(z)$ on another table.

Now, characteristic functions do a similar thing for probability distributions. Suppose $X$ has a distribution $f$ and $Y$ has a distribution $g$, and $X$ and $Y$ are independent. Then the distribution of $X+Y$ is the convolution of $f$ and $g$, $f * g$.

Now the characteristic function is an analogy of the "logarithm table trick" for convolution, since if $\phi_f$ is the characteristic function of $f$, then the following relation holds:

$$ \phi_f \phi_g = \phi_{f * g} $$

Furthermore, also like in the case of logarithms,it is easy to find the inverse of the characteristic function: given $\phi_h$ where $h$ is an unknown density, we can obtain $h$ by the inverse Fourier transform of $\phi_h$.

The characteristic function converts convolution to multiplication for density functions the same way that logarithms convert multiplication into addition for numbers. Both transformations convert a relatively complicated operation into a relatively simple one.

Improve this answer
$\endgroup$
5
  • 23
    $\begingroup$ Other items worth mentioned: (a) Recovery of moments via differentiation, (b) the fact that all distributions have characteristic functions (as compared to moment-generating functions), (c) The (essentially) one-to-one correspondence between distributions and their characteristic functions, and (d) the fact that many relatively common distributions have known characteristic functions but no known expression for the density (e.g., Levy stable distributions). $\endgroup$
    – cardinal
    Commented Apr 17, 2011 at 2:02
  • 4
    $\begingroup$ Good comments, @cardinal. Please consider turning them into an actual reply. $\endgroup$
    – whuber ♦
    Commented Apr 17, 2011 at 16:55
  • 1
    $\begingroup$ For those of you who understand this topic, is it at all related to Characteristic Equations, as used with recurrence relations (i.e. in Knuth's Concrete Math)? My guess is that they're very different and only share the word "characteristic" by chance, but I thought I'd ask. $\endgroup$
    – Wayne
    Commented Apr 18, 2011 at 16:35
  • 2
    $\begingroup$ @Wayne you should post this as a question. I think there is a close connection: Characteristic functions arise from Fourier Transform, which is the Gelfand Transform related to distributions on the real line. The Characteristic Equation of a recurrence relation seems to arise from the probability generating function, which is the Gelfand Transform associated with the natural numbers. The variables in recurrence relations can be thought of as taking values on discrete time-steps, i.e., natural numbers. $\endgroup$
    – cantorhead
    Commented Jun 14, 2015 at 14:10
  • 1
    $\begingroup$ @Wayne ... So I think the operator that takes a variable in a recurrence relation to its characteristic equation can be thought of as the "Fourier Transform" related to distributions on the natural numbers. I searched and didn't find this question but I would be very interested to see answers if you did post it. $\endgroup$
    – cantorhead
    Commented Jun 14, 2015 at 14:12
7
$\begingroup$

@charles.y.zheng and @cardinal gave very good answers, I will add my two cents. Yes the characteristic function might look like unnecessary complication, but it is a powerful tool which can get you results. If you are trying to prove something with cumulative distribution function it is always advisable to check whether it is not possible to get the result with characteristic function. This sometimes gives very short proofs.

Although at first the characteristic function looks unintuitive way of working with probability distributions, there are some powerful results directly related with it, which imply that you cannot discard this concept as a mere mathematical amusement. For example my favorite result in probability theory is that any infinitely divisible distribution has the unique Lévy–Khintchine representation. Combined with the fact that the infinitely divisible distributions are the only possible distribution for limits of sums of independent random variables (excluding bizarre cases) this is a deep result using which central limit theorem is derived.

Improve this answer
$\endgroup$
4
$\begingroup$

The purpose of characteristic functions is that they can be used to derive the properties of distributions in probability theory. If you're not interested in such derivations you do not need to learn about characteristic functions.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I suppose I might be interested in such derivations - I just don't quite get why we need to go to the characteristic function? Why is it easier than dealing directly with the pdf/cdf? $\endgroup$
    – Nick
    Commented Apr 16, 2011 at 21:10
  • 1
    $\begingroup$ @Nick This has a bit of folklore element, like "this is so elegant that this is THE representation of some distribution concept,...". Of course it helps with some maths, so it is not just a redundant toy, but for an everyday use it corresponds to a physicists forcing $\hbar$ to a classic problem just to use fine-structure constant. $\endgroup$
    – user88
    Commented Apr 16, 2011 at 23:41
  • $\begingroup$ We don't need to use them. I said only that they can be used. Sometimes they give a quicker derivation, sometimes they're no help at all. Whether a derivation is 'easier' depends on what you already know - if you don't already know about characteristic functions it won't be easier. In some cases moment generating functions provide an alternative, and have a more direct interpretation. $\endgroup$
    – onestop
    Commented Apr 17, 2011 at 5:02
2
$\begingroup$

The characteristic function is the Fourier transform of the density function of the distribution. If you have any intuition regarding Fourier transforms, this fact may be enlightening. The common story about Fourier transforms is that they describe the function 'in frequency space.' Since a probability density is usually unimodal (at least in the real world, or in the models made about the real world), this doesn't seem terribly interesting.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Note: A potential editor claims that the "characteristic function is the inverse Fourier transform". $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 29, 2015 at 1:40
-1
$\begingroup$

The Fourier transformation is a decomposition of the function (non-periodic) in its frequencies. Interpretation for densities?

Fourier transformation is the continuous version of a Fourier series since no density is periodic no expression like "characteristic series".

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.