How to judge if a datapoint deviates substantially from the norm This is Statistics 101, but I'm not a statistician and so can't seem to find the right technical jargon to google.
My company collects data at discrete points through time. Today's datapoint is positioned somewhat differently to the others, and so we're having a debate about whether this is an accident of chance or indicative of an actual underlying effect. What side you're on depends on how you eyeball the data, but we need to be able to detect these going forward. It is essentially a question of threshold placement.
"Given a set of datapoints through time, how different does a given datapoint have to be before it can be considered anomalous?", and "How unlikely is a given deviant point to have occurred simply by chance?"
Is this a simple question of outliers or standard deviations? Does the question require some kind of model-fitting to be solvable? I was originally thinking in terms of p-values and hypotheses here - as in, assuming a null hypothesis that the suspect datapoint is just a product of chance, could we calculate the probability of this null hypothesis being true in light of the data?
I don't even need a complete answer here, just pointers in the right direction. There must be a better way to decide these things than eyeballing.
 A: I'm going to say something that's a bit down at the simpler end, in case you end up drowning in ARMA models (although that of course it the correct way to go).
A graph with a fitted line and some measure of variability on could improve your eyeballing. I drew this graph yesterday:

Using this code with ggplot2 in R:
ggplot(mydata, aes(x=Time2, y=Reception, group=1)) + geom_smooth() +
facet_wrap(~Prison) + opts(axis.text.x=theme_text(angle=90, hjust=1)) + geom_point()

There's loads of default options for the smoothing etc., so you don't have to use it out of the box like I have, you can set it up much more nicely than this. But you can see a couple of "outliers" (I use that word loosely) in Prison B straight away.
Or the forecast package in R is very useful too.
As I say, this is more the quick and dirty approach so caveat emptor and all that.
A: Outlier detection in time series encompasses a large body of literature.  First you would want to have a time series model that fit well to the data when there were no suspect observations.  If for example an ARMA model works you might assume that the noise distribution is Gaussian.  There are at least two types of outliers.  Fox defined them in a 1972 paper. The best source to start with on this subject is the latest edition of Barnett and Lewis' "Outliers in Statistical Data" published by Wiley.  They have a chapter on time series.  My 1982 paper with Downing took the approach of looking at influence functions for autocorrwlation.  Our idea is that if an observation had a big effect on one of more of the lagged correlations it would also affect the model parameters adversely.  Martin, Yohai and others defined influence functionals for time series in a different way that seems to have better theoretical justification but addresses the same issue .  Ruel Tsay, George Tiao and others have also published work on outliers in time series.  I am less familiar with that. But our colleague IrishStat can probably comment on that and more. In the process of improving his autobox software over the years IrishStat and his son Tom have invested time into keeping up on the literature about outliers and level shifts (sometimes called interventions) in order to make their product state-of-the-art.
Just like with outliers in data that are not time dependent any outliers that are detected using time series methods should be studied to see why they occurred. Were they measurement errors?  Maybe a change in the behavior of the process?  Maybe a temporary intervention (like the Federal Reserve changing interest rates as an example)?  The reason if it can be found will dictate how the outlier should be treated.
