How can I test if predicted value is statistically different from the corresponding observed value, accounting for sample size of the observed value? I'm trying to test whether a observed value is statistically different from its corresponding predicted value. The observed value is a rate of a particular healthcare treatment, the predicted is the expected rate given the value of the independent variables. 
My first idea was simply to have CIs around the predicted value, and concluding that if the observed value fell outside the CI it was statistically different. 
But, I want to eliminate the possibility that the observed value falls outside the CI by chance because  it is the value for a comparatively small organisation.
I'm 'essentially' trying to do what a funnel plot does to identify special cause variation, whilst comparing predicted and observed values. 
Does anyone have any ideas on how I can do this? 
 A: This is a wonderful question, because it illustrates a common misunderstanding of statistics.  Contrary to the sloppy way people tend to talk about statistical testing, there is no such thing as a "statistically difference" between two values --- the values are either different, or they are the same.  Statistical tests arise only when we are comparing an unknown thing to a known thing.  In that context, when users of statistical tests refer to a "statistically significant difference" this is actually a (highly misleading) shorthand for statistically significant evidence of a non-zero difference ---i.e., the quantifier of "statistical significance" applies to the evidence of a non-zero difference, not the magnitude of the difference.  That is, we compare an unknown variable to a known stipulated value, and we use statistical methods to see if there is sufficient evidence to reject the null hypothesis that they are the same.
Your question is also not entirely clear because you refer to the predicted value as being the same as the expected value.  Assuming you are using some kind of parametric model, these things are different; the expected value is a function of the unknown model parameters, and so it is unknown, whereas the predicted value is a function of parameter estimates, so it is known.  If you are referring to an unknown expected value then you can conduct a statistical test to see if this is equal to (or different from) a stipulated value.  If you are referring to a known predicted value then there is nothing to test --- either it is equal to the observed value or it isn't.  In the latter case, what you can do is to form a prediction interval and then see if the observed value fell within the interval or not.  If you construct an interval with a high coverage probability and the observed value falls outside that interval, then this suggests that there is something defective in your prediction method.
A: First, I recommend approaching this descriptively, not inferentially.  You can show the difference you describe (the regression residual, assuming your predictions come from regression) in the context of all the other residuals.  A straightforward way would be to plot residuals (Y-axis) vs. predicted values (X-axis).  You and your readers can then judge the extent to which the residual in question stands out from the rest.  If you prefer a univariate plot, you could create a histogram of the residuals for the same purpose; perhaps the point in question will show itself to be an outlier.  And you could calculate its Z-score in the sense of how many standard deviations from the mean it falls.
As to the sample size underlying each data point (reflecting size of health care organization):  this is a good issue to recognize.  If you were computing some statistic on all the residuals or all the observations, you might want to weight according to these underlying sample sizes.  But in examining one point at a time for its unusualness, these weights would be less of a help.
I advocate a descriptive, not an inferential, analysis because I can't think of a proper way to implement the latter.  The fact that no one else has suggested an upvoted way after  250+ views of this post may say something.  To obtain a p-value for a single prediction's difference from its corresponding observation, one would need to be able to specify the "rules" of the way chance (randomness) would operate, and then to show just how unusual the finding in question would be in light of those rules.  But are there rules for what chance "normally" produces when it comes to residuals?  One sees all sorts of patterns, and they differ by topic, scale of measurement, variable distributions, type of predictive approach, and level of predictive accuracy, if not more.  Even if you standardize or studentize, some analyses produce residuals as large as +/-2.5; others, as large as +/-5.  Moreover, residuals are the product of human modeling decisions and as such are perhaps by definition not a chance phenomenon. I see no good basis for saying, "By chance -- under a certain null hypothesis -- a difference as large as or larger than this particular difference would occur only __% of the time."
A: Firstly, I think I might have an answer for you, but take what I say with a grain of salt. I am relatively new myself. But to answer your Question.
You can compute your test statistics normally and compare the emerging statistic a t-distribution with n-#predicted_values degrees of freedom, not a normal distribution. That should hold as long your null hypothesis is normally distributed.
The usage of the standard error instead of the standard diviation could be used too. I know it is used in some calcualtions for test statistics, but I don't know if it is usable here.
