Usefulness of standard deviation/alternatives for highly variable measurements

Question

EDIT: SOME ADDITIONS TO CLARIFY ORIGINAL TEXT

If I remember correctly I heard some mention of standard deviation for precipitation means of sums is pretty useless due to the highly variable nature of preciptation quantities.

Let's say that climatologists have calculated standard deviations for means of sums of monthly precipitation for every month of the year for 30 years of measurements. A monthly sum equals the total amount that has fallen during that month. So a monthly sum equals one measurement in this case. So if you take the average of the month of july over 30 years you have 30 measurements. If the standard deviation of these mean values are bigger than the mean values themselves it tells us there is a relatively high spread in the dataset. This is another way of saying that the coefficient of variation is big. But what would be considered big in this specific case? Are these sizes of the coefficient of variation normal for this type data? Lets assume here that all the coefficients of variations are above 100 %. Probably an irrelevant question in this forum.

Now when the difference between the average values of two 30-year-periods are calculated, each period introduces its own standard deviation. And the resulting standard deviation for the difference would be even bigger than the largest standard deviation between each of the normal periods. I believe this is called error propagation (please correct me if this is the wrong english terminology). If the resulting standard deviation is bigger than the difference of the mean values, it means that the difference between the mean values may be very far away from the true value. In other words a pretty "non-accurate" mean in this case right, which for certain/quite many observations would yield fictitious differences of means?

Precipitation can vary greatly in some regions of the world, for example due to large scale weather fluctuations like ENSO or other natural variaton. So perhaps 30 years is to low for averaging precipitation data due to high variability in some locations. The World Meteorological Organization recommends averaging over periods of thirty years. And this is common practice. Of course there are weaknesses by doing so and deviations from this practice exist. For instance some claim that thirty years is too low for certain climatic parameters due to their variable nature. This kind of answers part of my own question here. But if the precipitation data is only available for 30 years, are there any alternatives to standard deviation that would be recommended/considered more useful?

I think I have heard some mention that precipitation data from different locations may have different have different distributions. However is standard deviation only useful/make sense for normal distributions?

As a sidequestion: would the mean value be more accurate, with lower coefficient of variation if one has one million or billion years of measurements of data, even when each data point (spread) is highly variable?

EDIT 2: SHORT VERSION

If the data is not normally distributed what does coefficient of variation above 100 % tell us? What are the alternatives for detecting variation, if the alternatives are better/equally good (this is especially attractive to know about if the coefficient of variation is useless in my case)? Looking for answers which preferably are relevant to above example. Links to relevant studies are highly appreciated. Answers/research that provide intuitive examples/explanations are also highly appreciated. Of course answers to the other questions also are appreciated.

Answer 1

Question: Usefulness of standard deviation/alternatives for highly variable measurements?

Standard deviation will tell you whether or not the measurements are highly variable, it's not that you use "standard deviation" to predict the weather, it's that you use standard deviation to tell you if the other value (for which the standard deviation is provided) can be relied on as a predictor.

Even that alone is no guarantee. Example: It rained on this date 100% for the past 100 years, will it rain today? Answer: There's a good chance, but if there are no clouds in the sky there's 0% chance. The standard deviation of a single value is not the certainty of a result.

A simple example is provided on J. Smith of SNU's webpage on standard deviation:

"Everybody knows that when it comes to climate and weather, there really is no difference between Oklahoma and Hawaii. What?!?!?! You mean you don't believe me? Well, let's look at the statistics (after all, this is a stat course). The average (mean) daily temperature in Hawaii is 78 degrees farenheit. The average daily temperature in Oklahoma is 77 degrees farenheit. You see...no difference.

You still don't buy it huh? Well you are indeed smarter than you look. But how about those numbers? Are they wrong? Nope, the numbers are fine. But what we learn here is that our measures of central tendency (mean, median and mode) are not always enough to give us a complete picture of a distribution. We need more information to distinguish the difference.

Well before we go any further, let me ask a question: Which average temperature more accurately describes that state? Is 78 degrees more accurate of Hawaii than 77 degrees is of Oklahoma? Well if you live in Oklahoma I suspect you decided that 77 degrees is a fairly meaningless number when it comes to describing the climate here.

...

Okay...so the mean temperatures were 78 for Hawaii and 77 for Oklahoma...right? But notice the difference in standard deviation. Hawaii is a mere 2.52 while Oklahoma came in at 10.57. What does this mean you ask? Well the standard deviation tells us the standard amount that the distribution deviates from the average. The higher the standard deviation, the more varied that distribution is. And the more varied a distribution, the less meaningful the mean. You see in Oklahoma, the standard deviation for temperature is higher. This means that our temperatures are much more varied. And because the temperature varies so much, the average of 77 doesn't really mean much. But look at Hawaii. There the standard deviation is very low. This of course means the temperature there does not vary much. And as a result the average of 78 degrees is much more descriptive of the Hawaiin climate. I wonder if that has anything to do with why people want to vacation in Hawaii rather than Oklahoma?

From: "Probabilistic Forecasting - A Primer" by Chuck Doswell and Harold Brooks of the National Severe Storms Laboratory Norman, Oklahoma:

"Probabilistic forecasts can take on a variety of structures. As shown in Fig. 0, it might be possible to forecast Q as a probability distribution. [Subject to the constraint that the area under the distribution always sums to unity (or 100 percent), which has not been done for the schematic figure.] The distribution can be narrow when one is relatively confident in a particular Q-value, or wide when one's certainty is relatively low. It can be skewed such that values on one side of the central peak are more likely than those on the other side, or it can even be bimodal [as with a strong quasistationary front in the vicinity when forecasting temperature]. It might be possible to make probabilistic forecasts of going past certain important threshold values of Q. Probabilistic forecasts don't all have to look like PoPs! When forecasting for an area, it is quite likely that forecast probabilities might vary from place to place, even within a single metropolitan area.".

Question: However is standard deviation only useful/make sense for normal distributions?

All that standard deviation will tell you about "highly variable measurements" is that they are highly variable, but you knew that already; if the standard deviation is very low you can rely more, but not absolutely, on historical measurements.

As a sidequestion: would the mean value be more accurate, with lower coefficient of variation if one has one million or billion years of measurements of data, even when each data point (spread) is highly variable?

Q: Mean more accurate with more data points?: Yes.

Q: Lower variation (standard deviation)?: No, not if the "data point (spread) is highly variable".

The "standard deviation" doesn't affect the accuracy of your calculation of the mean, regardless of the standard deviation you have equal mathematical skills and calculate both the mean and standard deviation equally well. It's that with a standard deviation (accurately calculated) the mean (or any other value) has less meaning when the standard deviation is large. It's a less useful predictor.

With a very low standard deviation any prediction based on a single value (for example, the mean) isn't 100% reliable.

Question: Looking for answers which preferably are relevant to above example. Links to relevant studies are highly appreciated. Answers/research that provide intuitive examples/explanations are also highly appreciated. Of course answers to the other questions also are appreciated.

- Understanding the difference between climatological probability and climate probability

• Using the Probability Forecast Distribution Tool

• Why do ENSO forecasts use probabilities?

• Probabilistic Forecasting - A Primer (repeat of link given above)

- Bayesian probability

"Bayesian probability is an interpretation of the concept of probability, in which, instead of frequency or propensity of some phenomenon, probability is interpreted as reasonable expectation representing a state of knowledge or as quantification of a personal belief.

The Bayesian interpretation of probability can be seen as an extension of propositional logic that enables reasoning with hypotheses, i.e., the propositions whose truth or falsity is uncertain. In the Bayesian view, a probability is assigned to a hypothesis, whereas under frequentist inference, a hypothesis is typically tested without being assigned a probability.

Bayesian probability belongs to the category of evidential probabilities; to evaluate the probability of a hypothesis, the Bayesian probabilist specifies some prior probability, which is then updated to a posterior probability in the light of new, relevant data (evidence). The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation.".

- Modern Forecasting Papers

• A method for preferential selection of dates in the Schaake shuffle approach to constructing spatiotemporal forecast fields of temperature and precipitation (April 2017) by Scheuerer, Hamill, Whitin, He, and Henkel.

• Probabilistic temperature forecasting based on an ensemble AR modification (6 Aug 2015), by Möller and Groß.

• Spatial postprocessing of ensemble forecasts for temperature using nonhomogeneous Gaussian regression (30 June 2014), by Feldmann, Scheuerer, and Thorarinsdottir.

That should get you started, each of those papers has citation links which lead to newer papers.

Answer 2

An attempt to answer your questions, please correct me if I'm wrong.

Standard deviation is only one particular way of measuring variation.5 The measure for variation should match your distribution or optimization strategy. You could be right that for a given experiment, it does not correctly measure the variation as intended. As Frank Harrell mentioned in his comment, SD assumes symmetry of the distribution.

From the perspective of (statistical) machine learning / deep learning: "Solving an optimization problem with respect to a function requires a mathematical tool called calculus of variations. (..) Different cost functions give different statistics."1 In this context the error is often measured via mean absolute error 2 and mean squared error 3, but others are possible (MAPE, sMAPE, ...), depending on what is being optimized.

Another interesting resource on pitfalls when comparing models: 4.

Regarding the side question: this would totally depend on the underlying distribution. It could be that 30 year of data is not even close to the distribution of the climate change in the bigger picture, or, for example, it could be that the underlying distribution is easily represented with a linear model through the means of the last five years. We can not really be sure. We can only draw conclusions when our model is a good enough representation of the underlying distribution.

Answer 3

First suggestion is to start thinking about the coefficient of variation. That is, about $\sigma/\mu$. Intuitively, if the standard deviation is 'big', that means very little when the mean is just as 'big'. Conversely, if the SD is 'small', e.g. $\sigma < 10^{-3}$ this could still represent a very noisy process if the mean is 'tiny' e.g. $\mu <10^{-6}$.

Second, it seems OP may have overlooked that the standard deviation of a mean scales like $\sigma/\sqrt{n}$. This could be due to lack of clarity in the question. In particular, it looks like OP is talking about 30 years of monthly measurements, so $n=30$ for each month? But then also, we have 12 months when calculating the average difference between periods, so $n \approx 30 \cdot 12 = 360 \implies \sqrt{n} \approx 20$. The initial 'sniff test' is, then, whether $\sigma/\mu>20$, that is whether the standard deviation is roughly 20 times the mean. If it is, we might start suspecting that there isn't enough data.

Third, OP is trying to reason from first principles about an empirical question. Without knowing, $\mu, \sigma$ or indeed the distribution of precipitation (possibly indexed by geography) it is not possible to draw any firm conclusion from the information given. However, it seems OP may be hoping to poke a hole in climate science on the basis of the 'highly variable nature of preciptation quantities' - sorry this is not possible without numbers.

Fourth, there are dozens of other questions to ask, e.g. heteroskedasticty across months, across periods, across geographical regions. Then there're issues around whether a normal model is appropriate.

Also, there are some points in the OP where it is not clear what is meant, e.g. 'due to error propagation'.

Answer 4

I have had benefited from thinking simple and basic when I had a lot of questions.

Mean and standard deviation are both measures about a distribution. They simplify the distribution into something more understandable for us (We humans cannot really handle even a hundred data points). Standard deviation is calculated by assigning a cost from deviations from the mean. The cost function is usually (but not necessarily) a square function. Mean and std. dev. calculations themselves do not assume anything about distribution of the variable, that is the variable could be non-normally distributed as well and you can perfectly calculate std. dev. The inference of std. dev. on the other hand, depends on the distribution.

Let me clarify at this point one thing that is not clear in your question: The magnitude of standard deviation does not indicate whether the variable is distributed normally or not. You could have normal distributions with high variance. (High variance simply changes the scale of the axes in a plot, if you think in terms of coordinate system.)

Things get a little more complicated when you think in terms of samples and populations. Most of the time, we try to infer something about the population (e.g. precipitation levels) from what we have, the sample. The problem here is that we could be observing a randomly biased sample of the population just because of the random error component that cannot be modeled. We want to guard against it and use central limit theorem (clt) here. This, for instance, involves std. dev. calculations even for non-normally distributed variables.

The interesting thing with clt is that it shows a transformation of your data could be normal, even if your data is not. In clt, it is the sum (or mean) of observations as n gets large that is distributed normally.

So, non-normality is not a problem in inferring about the population. The high-variation problem you are referring to is only relevant here. With high-variation, you are less sure about the estimates you have and if it is high enough you can't differentiate your estimates with another. Intuitively, you are observing only some noise (possibly of a grand process).

Another thing you need to consider is that your data has a time-series component. (It is not something like, say IQ distribution.) Your data specifically marks how some measure changes over time. Non-normality is a common characteristics of time-series. Also, you need to be sure that the process is stationary.

As in the case of clt, a transformation of your data could (would?) be normal. Commonly, methods like differencing and log transformations are used.

tldr: There is nothing wrong with std. dev. calculations and its usefulness in non-normal data. Std. dev. says different things for different distributions. High std. dev also has an effect in interpretation. Try transformations if you need normality.