# standard deviations for mean vs for predictions

I work in finance and wanted to ask a quick question regarding standard deviation of data sets. I have collected data sets over the last 10 years and would like to use them to make a prediction for the coming year. The question I want to answer is, what the value that has a 95% probability of not being exceeded is? As we know, this is exactly $$\overline{x}+2\sigma_{\overline{x}}.$$Each of the data sets (from every year 365 datas) has about the same standard deviation. Now we learned in university that the standard deviation of the mean is just $$\sigma_{\overline{x}}=\frac{\sigma}{\sqrt{n}}.$$But I then get an estimate for next year with a standard deviation that is much too low and thus a maximum value that is much too low in contrast to recent years. In my opinion, this should be the approximatly the same standard deviation. Do I simply have to take the mean value for the standard deviation for predictions? What is the difference?

As noticed in the answer by Peter Pang, you are assuming normal distribution here. Are you sure this assumption holds? If it doesn't hold, $$\mu + 2\sigma$$ does not give you the 95% percentile. If you can assume that there are no changes over time (unlikely), then the answer is just to compute the 95% percentile from the data. If it changes, you can use something like quantile regression, a time-series model that makes distributional prediction, or a probabilistic model.

I believe there are quite a lot of assumptions made and confusions.

So first, you have assumed the data $$x$$ is normally distributed, therefore, $$x \sim \mathcal{N}(\mu, \sigma^2),$$ where $$\mu$$ and $$\sigma^2$$are the mean and the variance. But such an assumption is very likely to be wrong, and I would encourage you to make a histogram and see how are the data distributed.

And based on that you want to obtain the 95% interval of $$x$$ in the coming year. With the distribution of $$x$$ assumed, it can be inferred as follows . If assuming the distribution is normal, the final distribution for $$x$$ is given by (using Bayesian approach) \begin{aligned} p(x_{n+1}\vert\{x_i\}) &= \int d\mu d\sigma^2 \mathcal{N}(\mu, \sigma^2)p(\mu, \sigma^2\vert \{x_i\}), \end{aligned} where $$p(\mu, \sigma^2\vert\{x_i\})$$ is the posterior on the mean and the variance with all the previous samples given, which is given by $$p(\mu, \sigma^2\vert \{x_i\}) \propto \prod_i \mathcal{N}(\mu, \sigma^2)(x_i)p(\mu, \sigma^2),$$ where $$p(\mu, \sigma^2)$$ is the prior.

With that calculated, the 95% credible interval can be inferred.

One last thing, the equation you quoted is regrading the confidence interval of the mean $$\mu$$ based on the sample mean $$\bar{x}$$, not an interval for $$x$$.

• In a frequentist setting: if $\bar X$ is the mean of a normal sample of size $n$ from a population of unknown mean $\mu$ and variance $\sigma^2,$ then a 95% confidence interval for $\mu$ is of the form $\bar X \pm t^*S/\sqrt{n},$ where $S$ is the sample standard deviation and $t^*$ cuts probability $0.025$ from the upper tail of Student's t distribution with degrees of freedom $\nu = n-1.$ // A 95% confidence interval for an additional observation $X_{n+1},$ is of the form $\bar X\pm t^* S\sqrt{1+\frac 1n},$ which takes into account the est. variances of both $X_{n+1}$ and $\bar X.$ Commented Mar 15, 2022 at 22:57
• @BruceET As someone from the Bayesian side, I am not familiar with the calculation of confidence. Would you mind point me to the derivation of the confidence interval for $X_{n+1}$? Commented Mar 15, 2022 at 23:43
• See this part of Wikipedia page on prediction intervals. Commented Mar 16, 2022 at 7:35