9
$\begingroup$

How bad are the last 11 dry years in a row?

In the following graph there are 61 years of records of the annual energy that could be transformed in electricity using water-driven power plants in Chile:

Graph 1

In order to explain there is a trend of dry years, I am comparing that the last 11 years are below the average of the previous 50 years with records, but to give a sense of how bad it is, I would like to take the probability of having these 11 years below the average in a row at the end just for chance (expecting it to be very low, so it tell we have indeed a trend), but I don't know how to do it, since the time variable should have a meaning: the problem is somehow like having 11 tails in a row after 50 random previous results on 61 coin tosses, different of just counting 11 tails out of 61 tosses, but since the average change with each year, it is different from the coin example (which are there plenty of examples in Google).

Here you could find the table with the data: Google Sheets, but I preferred as an answer, the explanation of how to get the mentioned probability than just the final number.


What I have tried so far (added later)

I have test the data visually with a Gaussian distribution by taking the data Empirical Distribution and its look its fit quite good:

Empirical cumulative distribution

So since it don't look skew, I think is a good assumption consider that there is a $50/50$ chance of being above or below the average. With this, the problem "looks" similar to having coin tosses, so the total amount of alternative results are $2^{61}$, and having $11$ results below the average at the end becomes a string of $\underbrace{2\cdot 2\cdots 2\cdot 2}_{50\,\text{times}}\cdot \underbrace{1 \cdot 1\cdots 1\cdot 1}_{11\,\text{times}}=2^{50}$ alternatives fitting the required pattern, so I could estimate the probability of having the shown results just by chance as: $$ P\biggr(\text{the last 11 years below the average in a row}\biggr) = \frac{2^{50}}{2^{61}} = 0.0488 \%$$

Where I used that having exactly the average is a zero-measure point so it can be ignored.

But the issue in my line of thought, I think, is that the time variable is not considered, and the obtained probability thinking in coin tosses is valid for 11 heads (or tails) in a row independently of where the row is located, so having it exactly at the end could lead to a probability even lower.

So far, I think the result could be used as an upper bound for the real probability, but since I don't know nothing about stochastic time series, I would like to see if what I have done make any sense or not.


Added later

After the comment by @TickaJules I realize the complications: a change in the average could drop my assumptions since I have implicitly assumed the process is stationary.

But in the same line, if I compare the same statistics for the previous 50 years, which looks quite Normal/Gaussian, with the final 11 years were both the average and the standard deviation have decreased, they don't look Gaussian at all, and even when they are too few points to say something meaningful, the first graph kind of show a decreasing tendency, which is somehow supported but their cumulative distribution skewed to lower values:

First 50 years

Final 11 years

So I think that the previous calculated probability could still be a valid calculation considering it as a measuring a probability of $\mathbf{99.95\%}$ chance that something structural have changed in the last 11 years compared with the previous 50 years... or it is just too "cherry picking" as an assertion?

My arguments are:

  • If I believe that nothing have change in the last 61 years, since the aggregated data looks Normal, then the procedure for taking the probability $p = 0.0488 \%$ is valid, so is far more probable that indeed something "bad" have happened - and ironically this dismissed the assumption that nothing have happened.
  • And if I believe that something have change in the last 11 years, since these dry years in a row skewed the data to lower values, thinking in a moving average, the average now is lower making even harder to these years to be below it, but indeed it still happening. Since all the data is been considered in the analysis, I think is not skewed enough to say it have so much weight to lower values now, than it is more probable to have lower values than above the average, even when it is indeed what is happening so far, which somehow supports the hypothesis that something structural is different now. Following this argument, then the probability $p = 0.0488 \%$ is an upper bound for the "true probability", so considering all the years registered I have that at least with a probability of $99.95\%$ something structural have change on the final 11 years.

Does it make sense? or there are still something missing in what could be explaining these 11 dry years?

$\endgroup$
14
  • 4
    $\begingroup$ Hi, there are many possible ways to answer this based on what assumptions you choose to make. This makes it a bit ambiguous for a math question. $\endgroup$ Commented Nov 28, 2022 at 1:29
  • $\begingroup$ @JairTaylor Could you elaborate about the assumptions you are thinking about?... Maybe I am not completely understanding what I am asking for, but my intention is based just on the available data, with the empirical distribution of those 61 samples, figure out the probability of having those 11 samples in a row that are below the average, or maybe below some fix value since I am comparing it with the average of the previous 50 samples instead of the overall average which change with each of the additional sample of the 11's row, as they are introduced as time passes in the time series. $\endgroup$
    – Joako
    Commented Nov 28, 2022 at 2:24
  • 2
    $\begingroup$ Well, how do you find the empirical distribution for those samples? You could assume a normal distribution, for example, or some other distribution. Or, you could take a bootstrapping approach and re-sample the previous 61 records. $\endgroup$ Commented Nov 28, 2022 at 3:02
  • 1
    $\begingroup$ I think it's reasonable as a null hypothesis to assume that there's a "true average rate" (stationarity) that you can estimate using all your data so then being "below average" is about 1/2 (non-skewness), then you can work out the calculation as a math problem. Also one could argue this assumption is at the root of all statistical evil. There are alternative models: regress to find the rate through time, but that seems to need more assumptions, etc., to be explained and backed up. $\endgroup$
    – TickaJules
    Commented Nov 29, 2022 at 2:31
  • 1
    $\begingroup$ In the first case (1) you need to figure out if the population means are different, which means the difference in the sample means is statistically significant (which has a precise meaning). And for (2) I meant - see if the year (as a variable) is a meaningful variable in the regression. Does knowing the year help you predict the energy value? If so, then you can argue there's a time element here and something is changing. So that involves looking at t-tests (not just beta hats). $\endgroup$
    – TickaJules
    Commented Nov 30, 2022 at 12:29

3 Answers 3

3
+50
$\begingroup$

plot of data

The last 11 years have very low ranks. If the data is such that every year is independently distributed, then there is a strong statistical significant effect that the recent 11 years are lower than the 50 years before that.

Asside from comparing the numbers above/below the mean (whose worst case give the p-value 0.000488) you could also use a rank test, which gives an even lower p-value.

    Wilcoxon rank sum test

data:  y[1:50] and y[51:61]
W = 492, p-value = 4.686e-05
alternative hypothesis: true location shift is not equal to 0

An important question is, "does it make sense to assume that the years are independent?".

Clearly the hypothesis that you have a steady state distribution where every year is independent is wrong. However, this does not need to mean that 'something structural have changed'. It can be that you have random fluctuations over larger time scales that influence multiple years. It can be normal to have longer periods of years that are high or low.

or it is just too "cherry picking" as an assertion

This is always a risk with observational studies. Black swans happen and will be cherry picked.

More data, experiments, and theory can improve your believes.

then the probability p=0.0488% is an upper bound for the "true probability", so considering all the years registered I have that at least with a probability of 99.95% something structural have change on the final 11 years.

The p-value indicates the probability of a type-I error (the probability of falsely rejecting the null hypothesis when it is actually true). It is not the probability that a certain effect is present.

$\endgroup$
10
  • $\begingroup$ thanks for answering. The probability I have used was $p=\frac{2^{50}}{2^{60}}=0.0488\%$ it is not the p-value, but instead what I think is the probability of having the last 11 samples below the average in a row at the ending years in 61 total points of a non-skewed distribution (under assumption the process is stationary). Does this change your analisis? $\endgroup$
    – Joako
    Commented Dec 11, 2022 at 22:42
  • 1
    $\begingroup$ @Joako depending on how you formulate it, it can be a p-value. But in any way, you can not inverse it and state that 100% - 0.0488% is the probability that the alternative hypothesis is true. $\endgroup$ Commented Dec 11, 2022 at 22:57
  • $\begingroup$ mmm... sure I am mistaken, but I did't use it as a p-value, but as probability of having this trend just by chance: assuming the process is stationary (don't know if iid is also required), the probability should be ok, and is so low, that $1-p$ should tell it is not due just for chance, which ironically says also that stationary wasn't a good assumption, so something should have change over a stationary mean and variance assumption. As you said, maybe there is a underlying trend, natural or man-caused, it is highly probable the process is not stationary, (...) $\endgroup$
    – Joako
    Commented Dec 11, 2022 at 23:26
  • $\begingroup$ (...) which I think somehow the $99.95\%$ is measuring... but I don't know how to proper measure there is a trend behind, or how to measure how non-stationary have been those last 11 years compared with the previous 51 registered years... I hope those numbers could be used as an upper bound, or maybe learn how to calculated a properly measure for showing there is a trend/change compared with the previous years. $\endgroup$
    – Joako
    Commented Dec 11, 2022 at 23:29
  • 2
    $\begingroup$ @usεr11852 I make those images on my mobile phone. $\endgroup$ Commented Dec 13, 2022 at 18:21
1
$\begingroup$

This will be a two-part answer, the first part a direct answer to the question, and the second part a commentary.

Part 1: A simple, and exact, way to do it is to use the Hypergeometric distribution, as follows.

I am going to translate your problem into an "urn" model. We have 61 balls, corresponding to the 61 years of observations. 32 of these balls are "above" the average, and 29 are "below" the average. If I choose 11 balls without replacement - corresponding to the last 11 observations - what is the probability that they are all "below" balls?

The probability is easily calculated using any number of stat packages, or a calculator, as approximately $0.01\%$.

Part 2: However, this isn't really telling you what you want to know, in a formal statistical sense. To see this, consider whether you even would have done this test had, say, 5 of the last 11 observations been below the average, or what test you would have done if it had been the last 9 observations below average instead of the last 11 observations. The fact that you observed what appeared to be a highly unusual result, then tested the significance of exactly that result, pretty much cancels out the value of the significance test - as it's based on "samples from a finite population that I think are highly unusual" rather than "random samples from a finite population" as the test calculation assumes.

In an informal sense, it's OK to say "I thought this was a highly unusual result, and it is!" But it shouldn't be cited as a formal statistical test result.

Edit in response to comments:

To lend support to the validity of the Hypergeometric, I've constructed a simple example in code. We have 61 observations, 29 of which are "below" and 32 "above". We randomly rearrange them a million times, count the number of times that the last 11 observations have $0, 1, 2, \dots, 11$ "below" values, and compare to what the Hypergeometric distribution tells us to expect:

obs <- c(rep("above",32), rep("below", 29))

p0_to_11 <- rep(0,12)

for (i in 1:1e6) {
  x <- sample(obs)   # randomly rearranges the elements of "obs"
  nbelow <- sum(x[51:61] == "below")
  p0_to_11[nbelow+1] <- p0_to_11[nbelow+1] + 1
}

p0_to_11 <- p0_to_11 / 1e6

plot(p0_to_11 ~ c(0:11), type="b", pch=16, lwd=2, col=2,
     ylab = "Probabilities & frequencies",
     xlab = "# of 'below' observations")
lines(dhyper(0:11,29,32,11) ~ c(0:11), type="l", lwd=2, col=1)

enter image description here

The red dots indicate the observed frequencies, and the black lines are the Hypergeometric probabilities. There would be red lines too, except that the black lines overlay them. This is at least supporting evidence for the statement that the Hypergeometric is indeed the distribution to use in this circumstance.

$\endgroup$
11
  • $\begingroup$ Thanks for answering. Maybe I am mistaken, but if I am not the Hypergeometric Distribution does not care about where the samples with the wanted characteristic are located within the data, so it doesn't take into account that the values here are in a row and also that the this row is located at the end, which reduce the amount of possible combinations that fulfill what those 11 samples below the average in a row at the ending years represent, giving an even lower probability. Hope you could comment if I am mistaken. $\endgroup$
    – Joako
    Commented Dec 11, 2022 at 21:13
  • 1
    $\begingroup$ I am sorry to say you are mistaken. If I select all 61 balls in a completely random order, the last, let us say, 11 selected are a sample drawn from the 61 balls. The distribution of the number of "below" balls in this sample of 11 is Hypergeometric. Similarly, the first 11 are a sample of 11 balls from the 61 (without replacement), and balls 11-22 are a sample of 11 balls... You get to choose which 11 balls you are using as the sample, and as long as that choice is independent of the ball characteristics, you'll end up with a Hypergeometric... $\endgroup$
    – jbowman
    Commented Dec 12, 2022 at 1:47
  • $\begingroup$ Don't confuse the mechanics of the sample selection process with the statistical properties of the sample selection process (it's easy to do!). The key here is that the last 11 balls wound up being the last 11 balls for reasons that have nothing to do with their values ("above" or "below"), at least under the null hypothesis. Whether or not I arrange the balls in a row and choose the last 11 or just dump the last 11 into a bag is irrelevant. $\endgroup$
    – jbowman
    Commented Dec 12, 2022 at 1:51
  • $\begingroup$ As a probability stated as the number of favourable combination over total combination, I think it is relevant, since there are much more combinations of 11 sample of a row at any place than 11 in a row at the end (which have only one position for those samples). This is why mi intuition says is relevant. $\endgroup$
    – Joako
    Commented Dec 12, 2022 at 2:28
  • $\begingroup$ No, there aren't more combinations of 11 in a row anywhere than at the end. Pick any 11; you have one combination that's all "below", 11 combinations that are ten "below" and one "above", 55 combinations that are nine "below" and two "above", and so forth. It doesn't matter whether they are the first 11, the last 11, the middle 11, or whatever. Arbitrarily choose 11, for example, the last 11, and the number of "above" and "below" balls has to add up to 11, and the possible combinations are the same. $\endgroup$
    – jbowman
    Commented Dec 12, 2022 at 15:11
0
$\begingroup$

The easiest excel-friendly way to show the trend is to do a polynomial interpolation,

a more sophisticated version of this would be to use a digital filter.

However, apart from the approach @TickaJules suggested, I don't think there is a sensible way to define or interpret the probability of $11$ specific below-average data points.

$\endgroup$
2
  • $\begingroup$ Thanks for taking the time to answer. I don't think an interpolation neither a moving average could show any trend with so few data. But I disagree with the interpretation of the 11 point below the average because this non-negligible fact: since the sample space is equilibratly weighted between samples above and below the average (it could be any other case, as a lognormal distribution as counter example), I think is valid to to think there is near a 50/50 chance to have a sample over or below the average. (...continues) $\endgroup$
    – Joako
    Commented Dec 2, 2022 at 3:24
  • $\begingroup$ (...) Then, the probability has meaning because it is not "the probability of having 11 samples below the average", but instead, the "the probability of having 11 samples in a row at the endbelow the average", as for the examples of rows of coin tosses it change things a lot. But the problem are about if the interpretation I am doing is valid, which I don´t know (but it makes sense to me), but also if the probability is properly obtained, since in some websites I find different answers (like this one) $\endgroup$
    – Joako
    Commented Dec 2, 2022 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.