Population or Sample Standard Deviation: monthly climate data

Question

I have daily mean temperature values for every day of every month for 90 years.

I want to figure out:

• "SD$_m$": the standard deviation of daily temps across all days in a given month, for each month

• "SD$_{yr}$": The standard deviation of monthly means for a given year, for each year.

My question: Do I use sample [s] or population [$\sigma$] standard deviations for each of these above calculations?

$$s = \sqrt{\frac{\sum_{i = 1}^N(x_i - \mu)^2}{N - 1}} $$

$$\sigma = \sqrt{\frac{\sum_{i = 1}^N(x_i - \mu)^2}{N}} $$

My goal is to compare SD$_m$ to SD$_{yr}$ via regression to gauge their relationship for developing a climate metric. My confusion comes from the fact that both metrics are calculated from the same data, but both metrics would qualify "population" differently (i.e., SD$_m$'s "population" consists of all days in each given month while SD$_{yr}$'s population consists of all months within a given year). Or am I thinking about this incorrectly??

Answer 1

You probably want the first equation. The difference between the two quickly gets into mathematical statistics and opinions (we don't all agree on the best way to estimate mean; for instance, look at the midhinge or trimean), but R calculates variance the second way (sd(X)). Python/NumPy calculates it the first way (np.std(X)), though, so it's a matter of style, but you'd probably want to use the first equation.

The first estimator is related to an unbiased estimator (see footnote), meaning that, on average, it estimates the correct population variance. The second estimator is biased but has a nice statistical property known as maximum likelihood.

However, both are instances of statistical inference, where the goal is to make a conclusion about some greater population, given that we have sampled some subset of that population. It's not clear to me that you're doing inference. The opposite would be descriptive statistics, where you're just describing the values you have. I would argue that you have the population under consideration, in which case, your second equation is the correct one to do since it aligns with the calculus of population variance (related to something called the second moment of a distribution). Nonetheless, your field probably uses that first equation you gave, as that is the standard deviation estimator given in freshman or AP statistics that so many people take and use as their statistics background.

Footnote: The estimator you've given is biased, but you're taking the square root of an unbiased estimator for variance. Amazingly, the square root of that unbiased estimator winds up being biased, but I suppose being the square root of an unbiased estimator is something.

Footnote 2: There are examples of silly estimators that are, in some sense, legitimate (the technical term is admissible). For instance, you could define your estimate as a constant function, saying that standard deviation is 3 no matter what data you collect, and that would be an admissible estimator since it gets the right answer when population standard deviation indeed is 3. So you see how the estimator to pick can be a matter of opinion!