# I have a very fuzzy data set, what can I do to 'smooth' it?

Can you suggest some pointers on how I can manipulate my data to be 'smoother'? Any algorithms, or techniques that would be useful in this aspect?

• The data does not follow a known distribution as far as I know.

• It consists of 10+ different time series, which might have some relation to each other.

• I have another sample time series, to which I am trying to maximize the correlation of this one to that. The most important thing is to maximize correlation and hopefully preserve the shape of the aggregated 10+ time series combined together.

• I have tried rectangular smoothing (3-point) and triangular smoothing (3-point) but they both seem worse than not smoothing at all. Triangular smoothing is however better than rectangular smoothing.

• I am reading something on Savitzky-Golay and am going to try that next, and see if I can obtain a better result.

• My question is - what other smoothing filter/techniques should I try?

• wld appreciate if anyone could recommend a gd reference book that's easy to understand too! – ool Sep 13 '12 at 5:56
• Reccomending a smoothing technique without some example dataset is like a medical doctor trying to prescribe vitamins to a random person. – Néstor Sep 13 '12 at 7:10
• I'm sorry it sounded harsh, but as @gung said, I was just trying to exemplify my problem trying to answer this question. Any answer will be probably too general for you to use, that's why we need an example, goals, etc. as gung already explained :-). – Néstor Sep 13 '12 at 16:23
• If you want to smooth data in order to aid visualisation, great. If you want to smooth in order to interpolate, great. But, if you want to smooth the data and then use the smoothed version for modelling this is in general not a good solution. Just use the original data without pre-filtering unless you know what you are doing and have a good reason for pre-filtering. – Bogdanovist Sep 14 '12 at 0:01

Manipulating data is not a term I would use. It is common for statisticians to apply kernel density estimation or other univariate density estimation techniques to create an estimate of the population probability distribution. This assumes that a density exists for the distribution.

Although the question could be made more precise and a data example would be helpful I think it is fair to give my answer. My assumptions are as follows:

1. The author has a data set which he believes is a sample from an absolutely continuous distribution and hence has a density (that presumably is a relatively smooth function).
2. When the data is plotted using a histogram it is not sufficiently smooth.

These are the conditions under which smoothing of the histogram to form a probability density estimate are appropriate. Kernel density estimation is one very common way to do this. The amount of smoothing is always an issue and requires judgement. For kenrel density estimation the kernel has both a shape and a bandwidth. As Silverman points out in his book, the bandwidth is far more important than the shape even though there is theory to tell you the optimal shape. Here is an amazon link to Silverman.

Since the data are time series the Fourier transform of the autocorrelation function is spectral desnity function. Given a sample from a stationary time series you can get sample estimates for the autocorrelation function. The Forier transform of the sample autocorrelation function is called the periodogram. Due to sampling error the periodogram can appear very noisy. Just like you can smooth a histogram with a kernel smoother to estimate a probability density, so too can you smooth a periodogram to get a smooth estimate of the spectral density.

The smoothed spectral density can be transformed back to the time domain using the inverse Fourier transforn to get a smooth version of the autocorrelation function. The autocorrelation function estimate can suggest a possible ARMA model to fit the seires in the time domian. Fitting a parametric model in the time domain is one way to filter out the random white noise component of the series.

Since you mention having ten different time series that may be related (we call such series as being cross-correlated meaning that X at time t is correlated to Y at time t+h for various lead or lag time units. There are multivariate time series models particular of the autoregressive moving average form that can be used for this. Parametric forms for these model can be identified using the autocorrelation and cross-correlation function estimates.

Good references for univariate time series and multivariate time series are Box and Jenkins and Tsay for univariate and mutivariate/multiple time series respectively. Actually Tsay's text includes univariate GARCH models which are important special models for financial time series as well as other univariate models and a substantial chapter on multivariate models. Reinsel is completely devoted to multivariate time series.

• thanks but i'm not estimating the density – ool Sep 13 '12 at 23:48
• I based my answer on your original question which made no mention of time series and the idea that this would be a multivariate time series problem makes me wonder how just saying manipulating data and smoothing would give anyone a clue as to what you are taking about. Certainly there are smoothing techniques that can be applied in the frequency domain. But maybe you should think about applying multivariate ARIMA type models. But please realize that fitting models and smoothing is not manipulating data! – Michael R. Chernick Sep 14 '12 at 0:16
• Oh by the way if you are doing univariate time series analysis in the frequency domain the same kernel density approach can be used to estimate the spectral density. – Michael R. Chernick Sep 14 '12 at 0:18
• i think i am doing it in the time domain. thanks for the tips. i certainly don't mean 'manipulate' my data but i def need to make it smoother. – ool Sep 14 '12 at 1:03
• perhaps you could edit your answer wrt time series data smoothing? wld really appreciate it. – ool Sep 14 '12 at 1:10

For smoothing you can use: Moving average filter, Savitzky-Golay filter, Gaussian filter, Kaiser window, Continuous Wavelet Transform, Discrete Wavelet Transform etc...

• seems like a good start, but which technique is likely to work in this case? could you possibly rank them in usefulness? i guess one of my goals would be to reduce the 'fuzziness' or the jumpiness around sharp changes, but still retain the overall shape of the data set. – ool Sep 13 '12 at 23:49
• Is this a serious answer or a shotgun approach where if you mention enough smoothing methods you will hit one that applies. Maybe they all could given the vagueness of the problem. The idea of smoothing is in general is to separate signal from noise and all filtering approaches attempt that. – Michael R. Chernick Sep 14 '12 at 0:25
• the thing is i am new to all of this. and i have no idea what is the best approach. as the commenters requested for more info, i have updated the qn accordingly. – ool Sep 14 '12 at 1:06