164

The difficulty with using histograms to infer shape While histograms are often handy and sometimes useful, they can be misleading. Their appearance can alter quite a lot with changes in the locations of the bin boundaries. This problem has long been known*, though perhaps not as widely as it should be -- you rarely see it mentioned in elementary-level ...


39

You have no basis to assert your data are normal. Even if your skewness and excess kurtosis both were exactly 0, that doesn't imply your data are normal. While skewness and kurtosis far from the expected values indicate non-normality, the converse doesn't hold. There are non-normal distributions that have the same skewness and kurtosis as the normal. An ...


37

If the data is normally distributed, the points in the QQ-normal plot lie on a straight diagonal line. You can add this line to you QQ plot with the command qqline(x), where x is the vector of values. Examples of normal and non-normal distribution: Normal distribution set.seed(42) x <- rnorm(100) The QQ-normal plot with the line: qqnorm(x); qqline(x) ...


33

We usually know it's impossible for a variable to be exactly normally distributed... The normal distribution has infinitely long tails extending out in either direction - it is unlikely for data to lie far out in these extremes, but for a true normal distribution it has to be physically possible. For ages, a normally distributed model will predict there is ...


31

There are plenty of distance measures between two histogram. You can read a good categorization of these measures in: K. Meshgi, and S. Ishii, “Expanding Histogram of Colors with Gridding to Improve Tracking Accuracy,” in Proc. of MVA’15, Tokyo, Japan, May 2015. The most popular distance functions are listed here for your convenience: $L_0$ or ...


31

"The test showed that it is likely that the population is normally distributed." No; it didn't show that. Hypothesis tests don't tell you how likely the null is. In fact you can bet this null is false. The Q-Q plot doesn't give a strong indication of non-normality (the plot is fairly straight); there's perhaps a slightly shorter left tail than you'd ...


30

Of course, why not? Here's an example (one of dozens I found with a simple google search): (Image source is is the measuring usability blog, here.) I've seen means, means plus or minus a standard deviation, various quantiles (like median, quartiles, 10th and 90th percentiles) all displayed in various ways. Instead of drawing a line right across the plot,...


29

A kernel density or logspline plot may be a better option compared to a histogram. There are still some options that can be set with these methods, but they are less fickle than histograms. There are qqplots as well. A nice tool for seeing if data is close enough to a theoretical distribution is detailed in: Buja, A., Cook, D. Hofmann, H., Lawrence, M. ...


28

Cumulative distribution plots [MATLAB, R] – where you plot the fraction of data values less than or equal to a range of values – are by far the best way to look at distributions of empirical data. Here, for example, are the ECDFs of this data, produced in R: This can be generated with the following R input (with the above data): plot(ecdf(Annie),xlim=c(min(...


25

The canonical paper here was Wilk, M.B. and R. Gnanadesikan. 1968. Probability plotting methods for the analysis of data. Biometrika 55: 1-17 and it still repays close and repeated reading. A lucid treatment with many good examples was given by Cleveland, W.S. 1993. Visualizing Data. Summit, NJ: Hobart Press. and it is worth mentioning the more ...


23

Some tools for checking the validity of the assumption of normality in R library(moments) library(nortest) library(e1071) set.seed(777) x <- rnorm(250,10,1) # skewness and kurtosis, they should be around (0,3) skewness(x) kurtosis(x) # Shapiro-Wilks test shapiro.test(x) # Kolmogorov-Smirnov test ks.test(x,"pnorm",mean(x),sqrt(var(x))) # Anderson-...


20

A Poisson distribution is discrete while a normal distribution is continuous, and a Poisson random variable is always >= 0. Thus, a Kolgomorov-Smirnov test will often be able to tell the difference. When the mean of a Poisson distribution is large, it becomes similar to a normal distribution. However, rpois(1000, 10) doesn't even look that similar to a ...


19

Here's much easier way to understand it: You can look at Binomial distribution as the "mother" of most distributions. The normal distribution is just an approximation of Binomial distribution when n becomes large enough. In fact, Abraham de Moivre essentially discovered normal distribution while trying to approximate Binomial distribution because it quickly ...


18

Densities can be hard to work with. Whenever you can, calculate with the total probabilities instead. Usually, histograms begin with point data, such as these 10,000 points: A general 2D histogram tessellates the domain of the two variables (here, the unit square) by a collection $P$ of non-overlapping polygons (usually rectangles or triangles). To each ...


16

You need to fit these binned data with some distributional model, for that is the only way to extrapolate into the upper quartile. A model By definition, such a model is given by a cadlag function $F$ rising from $0$ to $1$. The probability it assigns to any interval $(a,b]$ is $F(b)-F(a)$. To make the fit, you need to posit a family of possible functions ...


16

@NickCox has presented an interesting strategy (+1). I might consider it more exploratory in nature however, due to the concern that @whuber points out. Let me suggest another strategy: You could fit a Gaussian finite mixture model. Note that this makes the very strong assumption that your data are drawn from one or more true normals. As both @whuber ...


16

The fact that box plots provide more of a summary of a distribution can also be seen as an advantage in certain cases. Sometimes when we're comparing distributions we don't care about overall shape, but rather where the distributions lie with regard to one another. Plotting the quantiles side by side can be a useful way of doing this without distracting us ...


14

OK, I've extensively revised this answer. I think rather than binning your data and comparing counts in each bin, the suggestion I'd buried in my original answer of fitting a 2d kernel density estimate and comparing them is a much better idea. Even better, there is a function kde.test() in Tarn Duong's ks package for R that does this easy as pie. Check ...


14

If $X$ has density (pdf) $f$ then $1/X$ does not have density $1/f$ (which is not even a density). Indeed, the change-of-variables formula teaches us that $Y:=h(X)$ has density $$\bigl|{(h^{-1})}'(y)\bigr|f(h^{-1}(y))$$ when $h$ is a "nice" invertible transformation. I gave the formula for a general $h$, because $h^{-1}(y)=1/y$ when $h(x)=1/x$, and that ...


14

In the univariate case, box-plots do provide some information that the histogram does not (at least, not explicitly). That is, it typically provides the median, 25th and 75th percentile, min/max that is not an outlier and explicitly separates the points that are considered outliers. This can all be "eyeballed" from the histogram (and may be better to be ...


13

Denby and Mallows 2009 ungated linkprovide a nice approach called the 'diagonally cut histogram', and provide a function 'dhist' in their supplementary material (available at the above link). Here is the abstract: When constructing a histogram, it is common to make all bars the same width. One could also choose to make them all have the same area. ...


11

A histogram represents probability by area: In this figure, the white region (to the left of $x=1$) comprises half the area. The blue region comprises the other half. The boundary between them at $x=1$ is, by definition, the median: it splits the total probability exactly in half. The areas in the next figure are shaded with varying densities of black: ...


11

I think it is worth mentioning that a Poisson($\lambda$) pmf is the limiting pmf of a Binomial($n$,$p_n$) with $p_n = \lambda / n$. One rather lengthy development can be found on this blog. But, we can prove this economically here as well. If $X_n \sim \mathrm{Binomial}(n,\lambda/n)$ then for fixed $k$ $$ \begin{align} \mathbb P(X_n = k) &= \frac{n!}{k!...


11

+1 to @NickSabbe, for 'the plot just tells you that "something is wrong"', which is often the best way to use a qq-plot (as it can be difficult to understand how to interpret them). It is possible to learn how to interpret a qq-plot by thinking about how to make one, however. You would start by sorting your data, then you would count your way up from the ...


11

While it's a good idea to check visually whether your intuition matches the result of some test, you cannot expect this to be easy every time. If the people trying to detect the Higgs Boson would only trust their results if they could visually assess them, they would need a very sharp eye. Especially with big datasets (and thus, typically with increasing ...


11

This answer has undergone significant changes as I investigate the wikipedia page. I've left the answers largely as they were but added to them, so at present this forms a progression of understanding; the last parts are where the best information is. Short answer: the wikipedia page - and the OP's formula, which seems to have been the same - are simply ...


11

There appears to be a difference in terminology (as so often is the case with a discipline used in so many areas), so I'm not 100% sure, but I think they're referring to kernel density estimation, with a Gaussian kernel, but performed on binned data. [Edit: if someone familiar with how the term "Gaussian smearing" is used in physics - and how it would apply ...


11

Age can not be from normal distribution. Think logically: you cannot have negative age, yet normal distribution allows for negative numbers. There are many bell-shaped distributions out there. If something looks bell-shaped it doesn't mean that it has to be normal. There is no way to know for sure anything in statistics, including from which distribution ...


11

According to wikipedia, mutual information of two random variables may be calculated using the following formula: $$ I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } $$ If I pick up your code from this: [co1, ce1] = hist(randpoints1, bins); [co2, ce2] = hist(...


11

No -- except in the most superficial of senses (like, "they're unimodal"). For starters, those curves don't seem to even be density functions: ... since if that increase going toward the left continues beyond the left edge of the plot, it won't even have a finite integral. (If it doesn't continue increasing as you move left, it implies bimodality, so it ...


Only top voted, non community-wiki answers of a minimum length are eligible