0
$\begingroup$

I have a question relating to computing confidence intervals that doesn't seem to be correct for my data. I have sales data on a test group and control group that looks something like this (note - the numbers have been made up just to show the kind of data I have):

Date        Group1 Control (Group1 - Control) Standardized(Group1 - Control)
X-Jan-20XX  123.5     187.2   (123.5-187.2)       -0.05
X           200.9     97.8 
.            .         .          .                 .
.            .         .          .                 .
.            .         .          .                 .
Xn           Yn        Tn         Dn                Sn

I then compute the 95% confidence intervals on the standardized(group1 - control) using the standard method (shown below) to identify the line where we can safely assume the data should fall under.

enter image description here

The problem I have is shown through the image below, as you can see majority of the data falls outside of the confidence interval. It must be noted that my data is not normally distributed, however, when I standardize my data I get mean 0 and standard deviation of 1. Can someone help with solutions on this issue?

What I am suspicious of is the fact that my initial raw data is not normally distributed whilst the methods implemented are for normally distributed data. I believe this is why there are many data points that are not captured under the normal distribution. If this seems to be the problem, can someone suggest solutions?!?

Thanks!

enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ A mean of 0 and standard deviation of 1 is NOT what characterizes a normal distribution!! You can get any data set to have a mean of 0 and standard deviation of 1 by the z-score transform: $z_i=(x_i -\bar{x})/s$ for every data point $x_i$. $\endgroup$
    – Dave
    Commented Mar 2, 2020 at 11:45
  • $\begingroup$ Thanks, that has been removed - do you have any comments on the rest of the question? $\endgroup$
    – Saad Ibrahim
    Commented Mar 2, 2020 at 11:50
  • $\begingroup$ Welcome to the site! What do the gray points represent? Are you examining the difference in scores across two groups over time. Do you want to know why the difference in standardized scores lies outside of the confidence bands? $\endgroup$
    – Thomas Bilach
    Commented Mar 2, 2020 at 12:23
  • $\begingroup$ Thanks! The grey points are the plotted values of the column Standardized(group1 - control). Yes, I want to know why a majority of the standardized scores lie outside of the confidence intervals (the confidence intervals are generated based on the standardized data). I get the same results when I use raw values, the raw values are significantly outside the confidence interval. $\endgroup$
    – Saad Ibrahim
    Commented Mar 2, 2020 at 12:46
  • $\begingroup$ Shouldn’t your best guess of the difference in population means be $\bar{X}_{1} - \bar{X}_{2}$? Just want to make sure you are calculating the correct margin of error. $\endgroup$
    – Thomas Bilach
    Commented Mar 2, 2020 at 13:24

1 Answer 1

Reset to default
1
$\begingroup$

You mix up the standard-deviation of your data, and the standard-deviation of the mean of your data. The confidence interval you mention is for the mean of your data, and it is not an interval where you "can safely assume the data should fall under". It is - in loose tersms - the interval where you "can safely assume that the mean of your data falls under". While the mean of your data sample is a single fixed number, it is subject to uncertainty. This is because it is estimated from a finite amount of datapoints. The mean of the distribution that underlies the datapoints might in fact be (slightly) different. If you estimated the mean from a finite sample, and if your data is normally distributed, then you can compute the uncertainty of your mean via the formula $\sigma/\sqrt{N}$, and with this you can construct confidence intervals. This does however not mean that a certain fraction of your datapoints lie withing that confidence interval. If $N$ goes towards infinity, your confidence interval gets smaller and smaller - meaning that a smaller and smaller fraction of points lies within the confidence interval. The confidence interval tells you something else: if you redo your whole experiment (gather new data), and for each experiment compute the mean of your datapoints, how many of these means would lie within that confidence interval.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Sip that explains a lot! Given the information you have just stated, can I get some clarity on another related question? The points before the "test start date" show the values of sales with no treatment, whereas, the data points after are a direct consequence of sales values due to a predictive model that has been implemented. I want to define a line where we can say with a 95% certainty that values outside of this range are due to benefits of the predictive model and not due to luck/random variation? Would it make more sense to define confidence intervals of variance? $\endgroup$
    – Saad Ibrahim
    Commented Mar 2, 2020 at 13:16
  • $\begingroup$ Glad to help, if this answer solved your problem please mark it as accepted by clicking the check mark next to the answer. see: (meta.stackoverflow.com/help/someone-answers). Regarding your additional question: I am not sure that I understand it correctly. Do I understand it right that you want to define a confidence interval for your predictions? If so, this will depend on many things in your particular application, and not very related to the question in this thread. I would suggest that you open a new question with an exact problem description. $\endgroup$
    – Sip
    Commented Mar 2, 2020 at 15:05
  • $\begingroup$ It's not that I want confidence intervals for the predictions, I want confidence intervals for where we expect our data to be before the predictions. To put it into context, the test start date is 12/02/2020 but we have analyzed data for 12/01/2020-12/02/2020 to see how the data fluctuates without any treatment (in terms of predictions). I then make predictions to try and boost the sales, so I want a line to show where we expect the data to lie within if there where no predictions to see if the predictions provide any "additional" benefits. $\endgroup$
    – Saad Ibrahim
    Commented Mar 2, 2020 at 15:35
  • $\begingroup$ For this you need to estimate the distribution of your data (not the distribution of the mean). If you assume that your data normally distributed, you can compute the mean m standard-deviation sigma, and then compute intervals. For example m plusmin sigma covers 68.3%. Alternatively, you can compute empirical percentiles (e.g. the 5th and 95th percentile). $\endgroup$
    – Sip
    Commented Mar 2, 2020 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.