3
$\begingroup$

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it generates a frequency curve like in normal distribution, Poisson distribution, etc. What is smoothing? How is it done? I have heard about the 'filtering techniques' and 'removing noise' but I don't understand the terms. Googling yields things like Kalman filter, moving average filter, low-pass filter, kriging, convolution, etc but they are very abstract concepts for me. Is there any good resource to start learning about these things from scratch?

$\endgroup$
3
  • 2
    $\begingroup$ en.wikipedia.org/wiki/Kernel_density_estimation introduces a common technique in this area with many links. $\endgroup$
    – Nick Cox
    Feb 10, 2014 at 18:28
  • $\begingroup$ Thanks @NickCox. I have tried several online tutorials including the wikipedia page you suggested but I don't understand why every explanation is so technical. Everyone starts with describing histograms but then suddenly jump to mathematical descriptions of kernel density estimation. I haven't found a simple example describing how smoothing changes the original data and helps in identifying the general trend. Isn't there a simple, humanly understandable explanation somewhere? $\endgroup$
    – umair
    Feb 10, 2014 at 22:29
  • $\begingroup$ There is no omelette without breaking eggs: the idea of a density function does depend on a little calculus. I wrote a review on kernel estimation, which is at onlinelibrary.wiley.com/doi/10.1002/esp.1518/abstract but you might not have access to it. $\endgroup$
    – Nick Cox
    Feb 11, 2014 at 0:15

1 Answer 1

8
$\begingroup$

To start with the title question, the density function is (for a continuous random variable) a function that describes the relative probability of the values taken by the variable (it is not probability! the word relative is important, since the probabilities are all zero); loosely it's proportional to the probability of being within a very small interval containing the point.

So where the density is high, values are relatively more probable, and where it's low, values are relatively less probable.

Areas under the density, between values on the x-axis, do give probabilities of lying in those intervals.

Here's an example of a density function:

enter image description here

The height can be taken as relative probability in that the height at 2 (about 0.27) is saying that a value very close to 2 approximately twice as probable as a value very close to 4 (where the density is about 0.14). [At the risk of sounding repetitive, the density at 2 is not the probability of getting 2, because density is not itself probability. Indeed density can happily exceed one for example.]

You could work out the probability of getting a value between 2.5 and 3.5 by finding the area of the curve between those values (that area turns out to be 0.223 for that density).

See this description relating to an analogy which might give a clearer sense of why it's called 'density'.

When dealing with discrete random variables, such as counts (and this is where the Poisson you mentioned comes in), you're not dealing with density, but an actual probability function (sometimes 'probability mass function'). The height of the probability function is actually the probability of the given value.

You seem to be envisioning specific, named probability distributions as things obtained by manipulating something derived from data (smoothing a histogram).

Generally speaking a probability distribution such as the normal is a model, a hypothetical construct that is an approximation for the behaviour of some variable or variables.

Such simple models are not typically obtained by 'smoothing' a histogram or a polygon, but suitability (or more accurately, unsuitability) might be approximately assessed by considering such displays. Beware, however; using things like histograms - and by extension, frequency polygons - to assess distributions. Using them for this purpose must be approached with caution, as they can be misleading if taken too readily at face value. QQ plots and quantile plots or empirical cumulative distribution functions would often be a better way to spot the suitability of such models, though interpreting QQ plots in particular require a little more effort to learn.

See also this answer.

Smoothing of empirical distributions to produce approximate densities does happen (though kernel density estimation would be a more common way to do it than looking at frequency polygons, along with a few other techniques for producing smooth density estimates), but not, typically to identify a simple functional form. Usually such smoothing is done instead of identifying a specific functional form - one can deal with the smooth density estimate on its own terms, without naming it.

See here for a non-mathematical description of what a kernel density estimate is.

Here's a kernel density estimate, for data that doesn't come from a normal distribution, or a gamma distribution, or a lognormal, Weibull, beta, Burr, etc, etc:

enter image description here

As with a hypothetical density, these empirical density estimates give some assessment of the relative probability (assuming the density is smooth and doesn't vary too quickly)

Your later questions at the end there conflate a whole set of topics which are mostly unrelated to the initial question. You might do better with some books to start, rather than googling, which will lead you through a topic rather than exposing you to large numbers of not-closely related terms.

A possibly relevant post to a number of those topics with a number of links is here

$\endgroup$
2
  • $\begingroup$ Thanks @Glen_b. Am I confusing the 'smoothing' in time series data to remove noise with the 'smoothing' of frequency polygon to obtain a density function? Are they related or two completely different concepts with same names? My real goal is to understand kernel density estimation and eventually convolution which I am using to remove noise from traffic flow data. $\endgroup$
    – umair
    Feb 10, 2014 at 23:02
  • $\begingroup$ Those things are definitely related, but smoothing an actual frequency polygon is almost never done in practice when the desire is to produce a density estimate. (There's been some additions to my answer after your comment was posted - though not directly in response to it) $\endgroup$
    – Glen_b
    Feb 10, 2014 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.