5
$\begingroup$

We have financial some data (500-1000 samples), which is not normally distributed (well known fact from the literature). I have some ideas to do parametric transformations of this data (using some other data) to produce "adjusted" series. My goal is to find a transformation that makes the series normally distributed (with mean 0 and std deviation 1). What is the most appropriate statistic and corresponding test to optimize my parameters and determine if the outcome can be considered normally distributed?

Please also point me to an implementation, ideally in C/C++ or java.

$\endgroup$
3
  • 3
    $\begingroup$ Your transformation would need to be a bit weird in that it would need to pull the tails in -- preferably in a smooth way. But I'm not convinced of the advantage. What are you doing that you think you need a normal distribution? $\endgroup$ Jan 1, 2012 at 16:02
  • $\begingroup$ Your question leaves me unclear. You say you have financial data and refer to them as a "series". Are you doing time series analysis, and want a procedure and test to achieve stationarity? That's not quite the same as what you seem to be asking, and while you would need to get a stationary time series, as @PatrickBurns notes, it wouldn't necessarily need to be normally distributed. $\endgroup$ Jan 1, 2012 at 22:09
  • $\begingroup$ Regarding transformations, there is a very trivial one that will make your data exactly standard normal. You simply map the order statistics of your sample to standard normal quantiles. I highly doubt you'd want to do this though, as it loses a lot of information and doesn't really accomplish anything. $\endgroup$
    – dsaxton
    Jan 9, 2016 at 21:57

2 Answers 2

4
$\begingroup$

For financial data I have successfully used heavy-tail Lambert W x Gaussian transformations.

  • Pyhon: gaussianize is an sklearn-type implementation of the IGMM algorithm in Python.

  • C++: the lamW R package has an elegant (and fast) C++ implementation of Lambert's W function. This can be a starting point for a full C++ implementation of IGMM or MLE for Lambert W x Gaussian transformations.

  • R: the LambertW R package is a full implementation of the Lambert W x F framework (simulation, estimation, plotting, transformation, testing).

As an illustration consider the SP500 return series in R.

library(MASS)
data(SP500)
yy <- ts(SP500)
library(LambertW)
test_norm(yy)

SP 500 returns

## $seed
## [1] 516797
## 
## $shapiro.wilk
## 
##  Shapiro-Wilk normality test
## 
## data:  data.test
## W = 1, p-value <2e-16
## 
## 
## $shapiro.francia
## 
## 	Shapiro-Francia normality test
## 
## data:  data.test
## W = 1, p-value <2e-16
## 
## 
## $anderson.darling
## 
##  Anderson-Darling normality test
## 
## data:  data
## A = 20, p-value <2e-16

As is well-known, financial data typically have fat tails and are sometimes negatively skewed. For the SP500 case, skewness is not too large, but it exhibits high kurtosis (7.7). Also several normality tests clearly reject the null hypothesis of a marginal Gaussian distribution.

Since we only have to deal with heavy tails, but not skewness, let's fit a heavy-tailed Lambert W x Gaussian distribution using a method of moments estimator (one could also use the maximum likelihood estimator (MLE) with MLE_LambertW()).

# fit a heavy tailed Lambert W x Gaussian
mod <- IGMM(yy, type = "h")
mod

## Call: IGMM(y = yy, type = "h")
## Estimation method:  IGMM 
## Input distribution:  Any distribution with finite mean & variance and kurtosis = 3. 
## mean-variance Lambert W x F type ('h' same tails; 'hh' different tails; 's' skewed):  h 
## 
##  Parameter estimates:
##    mu_x sigma_x   delta 
##    0.05    0.72    0.16 
## 
##  Obtained after 4 iterations.

The heavy tail parameter $\widehat{\delta} = 0.16$ is significantly different from zero and implies heavy tails. For such a $\delta$ moments up to order $1 / \widehat{\delta} = 6.29$ exist.

The model check question is of course if the back-transformed data does indeed have a Gaussian distribution. Let's check again using test_norm():

# transform data to input data (which presumably should have Normal distribution); use return.u = TRUE to get zero-mean, unit variance data
xx <- get_input(mod, return.u = FALSE)
test_norm(xx)

Gaussianized SP 500 return series

## $seed
## [1] 268951
## 
## $shapiro.wilk
## 
##  Shapiro-Wilk normality test
## 
## data:  data.test
## W = 1, p-value = 0.2
## 
## 
## $shapiro.francia
## 
## 	Shapiro-Francia normality test
## 
## data:  data.test
## W = 1, p-value = 0.2
## 
## 
## $anderson.darling
## 
##  Anderson-Darling normality test
## 
## data:  data
## A = 0.7, p-value = 0.07

I think the plot and normality test results speak for themselves.

The package also provides a single function that does all these steps at once: Gaussianize() (this is also what the Python package implements).

$\endgroup$
2
  • $\begingroup$ It appears to me that the graph of your transformed data speaks volumes suggesting non-normality as the variance is clearly non-constant. Secondly your emphasis should be on generating a model which delivers normal errors as all the assumptions are about the errors not the deviations of observed values from a mean. Fat tails are easily handled by Intervention Detection flagging unusual i,e, non-predictable values. $\endgroup$
    – IrishStat
    Jan 10, 2016 at 0:53
  • $\begingroup$ My example is just a direct response to OPs question on how to transform data to something that's Normal. OP also mentioned he is working with financial data, so I used the SP500 example to illustrate the marginal distribution transformation. It is well known that financial data are not iid, but heteroskedastic and you can also fit a GARCH or SV type model to it to obtain (approx) normally distributed residuals. However, that was not what the OP was asking for -- so I didn't elaborate on this time series model fitting exercise. But in principle you are right. $\endgroup$ Jan 10, 2016 at 1:04
3
$\begingroup$

It appears that you are just asking for a test for normality. If so, Shapiro-Wilk is hard to beat. This is not, however, the easiest test in the pantheon to implement.

Why not just use R? The shapiro.test function will do the work for you.

$\endgroup$
4
  • 1
    $\begingroup$ the original poster is asking about how to transform data to normality, not how to test for normality. $\endgroup$
    – Macro
    Jan 1, 2012 at 22:41
  • 1
    $\begingroup$ @Macro - "What is the most appropriate statistic and corresponding test to optimize my parameters and determine if the outcome can be considered normally distributed?" This isn't quite asking how to do the transform, it's instead asking how to evaluate whether the transform is effective, i.e., (in this case) a statistic and corresponding test for normality. $\endgroup$
    – jbowman
    Jan 2, 2012 at 1:12
  • $\begingroup$ I don't know R :( And I have to do quite a lot of other computations in this program. $\endgroup$
    – Grzenio
    Jan 2, 2012 at 15:04
  • $\begingroup$ @Grzenio - I'll try to pull something together for you later today, probably in C/C++. $\endgroup$
    – jbowman
    Jan 2, 2012 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.