# How to find standard/linear deviation of time based data?

1416668402352, 10
1415684102290, 20
1415684402210, 30
1415684702188, 15
1415684702780, 25
1415685001845, 40
1415685301846, 100
1415685602110, 35
1415685603483, 50
1416668101756, 29
1415685901549, 95
1415686201903, 18
1415686502076, 24
1415686502765, 90
1415686801514, 32


In above data, first value is timestamp and second is value of something at specific timestamp.

By using this data I want to calculate value which is near to other values. I don't want to consider spikes in data (such as 100,90,95) which are occur due to some abnormal situation.

As a novice to stat, I don't know what is called for this sampling method??

can anybody provide me some algorithms/methods to find above stated value??

You will have to define what values you do consider spikes in the data. This can be defined either manually (by defining a manually fixed threshold) or using a proportional value to the standard deviation of data.

For this second case you could find the standard deviation of your vector x (std(x) for matlab) and then filter the vector by using only the values that are lower than the mean plus n standard deviations.

Something like: x = x(x<mean(x)+n*std(x))

Now, what do you actually want to achieve with the timestamp? I understand you want for a new input (time,value) to find other (time,values) that are similar to it. Given you do not have an specific classification for this different 'groups' you might want to follow an unsupervised learning approach to this problem. A simple solution (efficient but not always accurate) would be to use a clustering algorithm. A simple K-means algorithm will give you K-clusters, and you could calculate roughly the degree of belonging to this group as the distance to the centroid (for a more accurate belonging you will need to make a more sophisticated calculation or use fuzzy clustering).

A simpler solution would be to calculate the K-Nearest-Neighbors and there you will have a natural feature space grouping. This neighbors won't vote for an specific category to your data but just give you an idea of which other values are closer to to your input spatially.

Unless you want a more specific output, like constraining for time-ranges, this will give you a rather "natural" approach (spatially).

• infact I want to find threshold from data given in example. Dec 9, 2014 at 14:24
• Ok, well mean(x)+n*std(x) will be a consistent threshold in general. You will need to define n according to your problem. Dec 9, 2014 at 14:25