5
$\begingroup$

Is there any way to test whether a series should be logged or transformed in another way?

I have a code of which i use to run lots of different data through to forecast. Some of the data definitely need transforming however some don't. As the code has been written to be fully automatic it will be used by non-statisticians within the company so they will have no idea whether they should change the code to transform the data depending on the series. So i need tests which will check that for them and apply the transformation accordingly.

Here is a example data set that you can use:

M <- matrix(c("08Q1", "08Q2", "08Q3", "08Q4", "09Q1", "09Q2", "09Q3", "09Q4", "10Q1", "10Q2", "10Q3", "10Q4", "11Q1", "11Q2", "11Q3", "11Q4", "12Q1", "12Q2", "12Q3", "12Q4", "13Q1", "13Q2", "13Q3", "13Q4", "14Q1", "14Q2", "14Q3",  5403.676,  6773.505,  7231.117,  7835.552,  5236.710, 5526.619,  6555.782, 11464.727,  7210.069,  7501.610,  8670.903, 10872.935,  8209.023,  8153.393, 10196.448, 13244.502,  8356.733, 10188.442, 10601.322, 12617.821, 11786.526, 10044.987, 11006.005, 15101.946, 10992.273, 11421.189, 10731.312),ncol=2,byrow=FALSE)
Nu <- M[, length(M[1,])]

I have found boxcoxfit() from the package geoR finds the lambda for transformation....does anyone know how accurate this is for transforming the data?

ml <- boxcoxfit(Nu)
  Fitted parameters:
    lambda    beta  sigmasq
      0.59  375.43  3649.39
N<- ((Nu^(ml$lambda))-1)/ml$lambda
Improve this question
$\endgroup$
9
  • $\begingroup$ Perhaps heretical here, but an eyeball glance at your data suggests approximately linear trend, variability around it approximately constant, and obvious seasonality with peaks consistently in quarter 3. It's not obvious to me that any transformation helps at all. The fact that this is a rather short time series underlines that caution. This is all quite orthogonal to your request to automate analysis, naturally. $\endgroup$
    – Nick Cox
    Commented Oct 29, 2014 at 20:01
  • $\begingroup$ @NickCox thankyou for your comment, i have many different types of data sets, some which give similar results to this and some that don't. e.g transactional and non-transactional data, i have found by analysing them that some definitly need transforming, however we are in the process of linking my code to a database in which data will automatically change and many data sets will be run. I need to find a way that decides if it should be transformed and how without me even looking at the data itself. $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 30, 2014 at 9:30
  • $\begingroup$ Indeed. But this thread implies that the task is highly challenging as even for a simple data series (a) different software yields different suggestions (b) different analysts don't agree. $\endgroup$
    – Nick Cox
    Commented Oct 30, 2014 at 9:40
  • $\begingroup$ Yes i can see that... how inaccurate would the results be if i logged/transformed all the data sets straight away or just left them as they were? $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 30, 2014 at 9:49
  • $\begingroup$ It really does depends on the data and the decision. Any way, what does "inaccuracy" mean here? The key point is that the marginal distribution of a response variable may have little or nothing to do an appropriate time series model. @IrishStat's AUTOBOX is an attempt to build decisions into the program, so in effect the program designer is making your decisions for you. Looking at threads on this forum show that people disagree about strategy and style for time series just like anything else. $\endgroup$
    – Nick Cox
    Commented Oct 30, 2014 at 10:20

2 Answers 2

Reset to default
7
$\begingroup$

As @Irishstat points out you could use boxcox power transformation, which is a more general transformation function which also includes log transformation. R's forecast package has a function called BoxCox.lambda and BoxCox, you could use these two functions and determine if your data needs transformation. if lambda is close to 1 then your data needs no transformation, else you data needs appropriate power transformation.

Using your data 

x <- ts(c(5403.676,  6773.505,  7231.117,  7835.552,  5236.710, 5526.619,  6555.782, 11464.727,  7210.069,  7501.610,  8670.903, 10872.935,  8209.023,  8153.393, 10196.448, 13244.502,  8356.733, 10188.442, 10601.322, 12617.821, 11786.526, 10044.987, 11006.005, 15101.946, 10992.273, 11421.189, 10731.312),frequency =4)

lambda <- BoxCox.lambda(x, method=c("guerrero"))
lambda
0.3855427

x.transform <- BoxCox(x,lambda)
plot(x.transform)

Using the box.cox lambda yielded a lambda value 0.3855. You could use this in the BoxCox function as shown above.

Let us know if you find this post useful.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Have you any comment on why this recommendation (0.386) is quite different from 0.59? $\endgroup$
    – Nick Cox
    Commented Oct 29, 2014 at 19:58
  • $\begingroup$ Box-Cox optimization requires a model. I guess if no model is specified then a simple mean model is used. Seems kind of silly to be talking about Box-Cox when no reasonable model is in place but that's just my opinion. Furthermore untreated outliers distort the Box-Cox optimization as it assumes no deterministic structure is present. $\endgroup$
    – IrishStat
    Commented Oct 29, 2014 at 20:28
  • $\begingroup$ Power transformations ..other than -1,-.5 ,0.0,.5 ,1.0 are nigh impossible to justify when presenting a possible solution to a client . In my opinion it is an overkill to use values like .386 or .59 ....but that's just my opinion. $\endgroup$
    – IrishStat
    Commented Oct 29, 2014 at 21:06
  • $\begingroup$ It doesnt have to be Box-Cox, thats just something i read about. $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 30, 2014 at 9:24
  • $\begingroup$ Thankyou forecaster. Just to clarify, when you say that if its close to 1 then it doesnt need a transformation...how close to 1 do you think it should be? 0.8? $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 30, 2014 at 9:25
2
$\begingroup$

Power Transformations found via a Box-Cox test http://onlinestatbook.com/2/transformations/box-cox.html are useful/correct when a linear relationship is found between the expected value and the variability of the model errors. It has little to do with the variability of the original series. The range of transformations is from none to a reciprocal. Care should be taken to account for pulse outliers as untreated they can distort the Box-Cox conclusions. Furthermore note that error variance may also change in discrete steps quite free of the expected value . The appropriate remedy in this case is to Generalized Least Squares or as it is often known as Weighted Least Squares.

You might look very closely at my response to Seeking certain type of ARIMA explanation

UPON RECEIPT OF DATA (enter image description here :some 27 quarterly observations starting at 2008 q1

The ACF of the original series suggests a fairly strong seasonal structure. AUTOBOX automatically identified a model enter image description here and shown here enter image description here which yielded an ACF of the error process suggesting model sufficiency enter image description here . The model includes an identified intervention at period 21 (2013 quarter 1 ) of the 27 observations. A plot of the actual and the cleansed highlights the anomaly.enter image description here The actual/fit/forecast graph is here enter image description here with forecasts here enter image description here. In summary there was no need for any variance stabilization transformation for this data set. The optimal box-cox coefficient requires a model and in this case is 1.0. If you don't specify a model as is possible with boxcoxfit then in the absence of a good ARIMA structure and the identified anomaly at period 21 you might then get a lambda like .52 which is probably the result of an incorrect model.

Improve this answer
$\endgroup$
10
  • $\begingroup$ so is this suitable to apply to any series even if it may not need transforming? $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 27, 2014 at 15:07
  • $\begingroup$ Yes because if the error process is free of any of the structure I referenced then the conclusion will be No Transform is required. $\endgroup$
    – IrishStat
    Commented Oct 27, 2014 at 15:09
  • $\begingroup$ ok great, so i can apply that to my code so everything automates :) thankyou :) $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 27, 2014 at 15:12
  • $\begingroup$ Just be careful that the error term from your model is free of ARIMA structure , Free of any Pulses/level-Step shifts , Local Time Trends and Pulses AND that you have verified t hat the model parameters are invariant over time OTHERWISE you may be pinning the tail on the wrong donkey . $\endgroup$
    – IrishStat
    Commented Oct 27, 2014 at 15:18
  • $\begingroup$ ....How do i do all that? :/ $\endgroup$
    – Summer-Jade Gleek'away
    Commented Oct 27, 2014 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.