0
$\begingroup$

I am dealing with a problem where the decision of applying a power/log transformation on time series data has to be done statistically. I know it can be done visually but I have to build model for multiple time series so visualizing it won't automate the problem. One of the approach I am following now is check for skewness and if skewness is out of the range of -1,1 then apply power transformation. Note: I don't wan't to apply Box cox if I don't have evidence it has to be applied.

EDIT: After a lot of research I found a solution. So one way to figure out if you should log the time series before analysis is to fit a linear regression model using time series data. And afterwards using fitted regression model's results run Goldfeld-Quandt Test. Using the result of Goldfeld-Quandt Test(test for checking heteroskedasticity) we can determine if it could be helpful to scale down the time series data.

$\endgroup$
1
$\begingroup$

So you can use a test for normality as you are currently doing where the skewness should ideally be between -0.8 and 0.8 but you should probably also test for kurtosis if you are going this way.

Alternatively, you can use an inferential test of normality such as Kolmorogov-Smirnov (scipy.stats.kstest). This will tell you if your data is significantly different from the standard normal distribution ( p > 0.05 indicates that the distribution does not significantly differ from the normal).

If the KS test suggests that the distribution of the data is not normal then you can use Box-Cox to transform it.

$\endgroup$
5
  • $\begingroup$ But the question here will be if a distribution is not normal does that call for power/log transformation in time series modeling. I forgot to mention but I tried kstest and it always(for 1000 different time series) gives me 0 p-value which could be because it's a strict normality test. That was the reason I went with skewness. And also my time series data has negative data in between because of which box cox can sometimes not be applied. P.S. I agree that my problem has a lot of other odds $\endgroup$ – Pranav Bahl Apr 5 '17 at 19:42
  • $\begingroup$ Whether the distribution requires transformation depends on how you are analysing the data and what assumptions you are using. If for instance, you are running ordinary least squares regression on the data then you have to assume the data is normally distributed. If you have count data and you are using a general linear model to account for a Poisson distribution then it's fine to have non-normal data (note that ktest is not defined for discrete data). Edit: The usual way to deal with negative values is to either use missing values or transform the points to be all positive before transforming $\endgroup$ – Eumenedies Apr 5 '17 at 19:57
  • $\begingroup$ Because I am running multiple models so I am not holding an assumption here. The only assumption I have is if I transform the data which is if I scale down the exponential behavior in time series will that be helpful. Now to figure out if time series data has some exponential behavior I wanted to come up with a automated way of figuring out if the data has some exponential pattern. And further if down scaling that pattern can help model to omit better predictions. Edit: I am dealing with financial data $\endgroup$ – Pranav Bahl Apr 7 '17 at 19:14
  • $\begingroup$ Every model you run will have assumptions and you need to know what they are. If you want to know if log transforming the data will improve the model performance you can model both and test the comparative performance using a performance metric such as RMSE. But that would indicate that the relationship between the response and the predictor is exponential which is different to saying that the data is exponentially distributed. Without knowing what models you are using I cannot really say more. $\endgroup$ – Eumenedies Apr 7 '17 at 20:53
  • $\begingroup$ Totally agree with your point. And infact I am doing what you mentioned later, running model with transformation and without. But my problem was if I can generally do smart transformations. Figuring out a test which can help me decide if log transformation has to applied to the data or not rather than applying on all. Edit: Just to add to that the models I am experimenting with are: R's auto arima, Pyflux arima, FB prophet and some personal algorithm. $\endgroup$ – Pranav Bahl Apr 10 '17 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.