I work in finance and wanted to ask a quick question regarding standard deviation of data sets. I have collected data sets over the last 10 years and would like to use them to make a prediction for the coming year. The question I want to answer is, what the value that has a 95% probability of not being exceeded is? As we know, this is exactly $$\overline{x}+2\sigma_{\overline{x}}.$$Each of the data sets (from every year 365 datas) has about the same standard deviation. Now we learned in university that the standard deviation of the mean is just $$\sigma_{\overline{x}}=\frac{\sigma}{\sqrt{n}}.$$But I then get an estimate for next year with a standard deviation that is much too low and thus a maximum value that is much too low in contrast to recent years. In my opinion, this should be the approximatly the same standard deviation. Do I simply have to take the mean value for the standard deviation for predictions? What is the difference?
2 Answers
As noticed in the answer by Peter Pang, you are assuming normal distribution here. Are you sure this assumption holds? If it doesn't hold, $\mu + 2\sigma$ does not give you the 95% percentile. If you can assume that there are no changes over time (unlikely), then the answer is just to compute the 95% percentile from the data. If it changes, you can use something like quantile regression, a time-series model that makes distributional prediction, or a probabilistic model.
I believe there are quite a lot of assumptions made and confusions.
So first, you have assumed the data $x$ is normally distributed, therefore, $$ x \sim \mathcal{N}(\mu, \sigma^2), $$ where $\mu$ and $\sigma^2$are the mean and the variance. But such an assumption is very likely to be wrong, and I would encourage you to make a histogram and see how are the data distributed.
And based on that you want to obtain the 95% interval of $x$ in the coming year. With the distribution of $x$ assumed, it can be inferred as follows . If assuming the distribution is normal, the final distribution for $x$ is given by (using Bayesian approach) $$ \begin{aligned} p(x_{n+1}\vert\{x_i\}) &= \int d\mu d\sigma^2 \mathcal{N}(\mu, \sigma^2)p(\mu, \sigma^2\vert \{x_i\}), \end{aligned} $$ where $p(\mu, \sigma^2\vert\{x_i\})$ is the posterior on the mean and the variance with all the previous samples given, which is given by $$ p(\mu, \sigma^2\vert \{x_i\}) \propto \prod_i \mathcal{N}(\mu, \sigma^2)(x_i)p(\mu, \sigma^2), $$ where $p(\mu, \sigma^2)$ is the prior.
With that calculated, the 95% credible interval can be inferred.
One last thing, the equation you quoted is regrading the confidence interval of the mean $\mu$ based on the sample mean $\bar{x}$, not an interval for $x$.
-
1$\begingroup$ In a frequentist setting: if $\bar X$ is the mean of a normal sample of size $n$ from a population of unknown mean $\mu$ and variance $\sigma^2,$ then a 95% confidence interval for $\mu$ is of the form $\bar X \pm t^*S/\sqrt{n},$ where $S$ is the sample standard deviation and $t^*$ cuts probability $0.025$ from the upper tail of Student's t distribution with degrees of freedom $\nu = n-1.$ // A 95% confidence interval for an additional observation $X_{n+1},$ is of the form $\bar X\pm t^* S\sqrt{1+\frac 1n},$ which takes into account the est. variances of both $X_{n+1}$ and $\bar X.$ $\endgroup$– BruceETCommented Mar 15, 2022 at 22:57
-
$\begingroup$ @BruceET As someone from the Bayesian side, I am not familiar with the calculation of confidence. Would you mind point me to the derivation of the confidence interval for $X_{n+1}$? $\endgroup$ Commented Mar 15, 2022 at 23:43
-
1