I have a dataset and for each hour there is 3 readings (sometimes missing and sometimes clearly an outlier). I am trying to find the mean of the entire dataset for the parameter. It has been suggested to me to take the mean of each hour and then the mean of the entirety of those means as this will help minimize the contribution of single outliers AND it was suggested to me to take the mean of the entire set and ignore the time.

Those two methods seem to have very similar results and similar standard deviations. However, if I trim out the outliers some datasets are significantly different than this (these are usually one observation that is more than 4 times the others etc and I think due to an observation error) so I think removing these outliers would be a good thing. So what is the advantage of a moving average? Is it inapropriately used here or am I misunderstanding it?

My dataset sort of looks like below:

Hour| observation

0 | 5

0 | 6

0 | 5.6

1 | .

1 | 4

1 | 4.8

2 | 5.1

2 | 5.4

2 | 498 .....

  • $\begingroup$ How did you get these data and what are they? It seems like measurement errors if they are "readings" of something. en.m.wikipedia.org/wiki/Outlier Given the variance of the seemingly "correct" observations "4 times" seems too much. $\endgroup$ May 31 '15 at 0:48

Removing 'outliers' is a good idea IF the outliers are properly described as erroneous readings, mistakes or they are known to be misleading. However, to ascribe one of those properties to the outliers requires something other than just inspection of the data. Were the three readings at each time point intended to allow assessment of whether the measurement 'worked' or are they intended simply to reduce the influence of measurement variability? If it was the former then you should be considering the known properties of the test and measurement system when deciding what causes outliers. Missing values and odd things like the decimal point without numerals may be indicative of a 'fragile' measurement or recording system that can be expected to yield outliers that are mistakes.

If a four times range for repeated measurements is plausibly just a result of the underlying variability of the values or of the measurement method then do not remove them. Perhaps the median of the distribution, which is less strongly affected by outliers, will serve instead of the mean.

If you have lots of data then you are in a good position to work out whether the outliers are bad values, because you can plot the distribution of the readings.

  • $\begingroup$ It's an instrument that is known to make lots of errors and have missing values. I am unsure and will set up a meeting with the researcher thank you. $\endgroup$
    – Jacob Ian
    May 31 '15 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.