How to make a johnson unbounded transformation to make my data more gaussian like? in python

Question

I am a novice in stats and I would like to transform my data (house prices) using a johnson unbounded distribution to look more gaussian. I looked at pandas transform() but I can't really understand johnsons u. parameters to apply a lambda. Could someone help me out doing this transformation in python? I have the parameters but don't know which is which to fit into the formula (or if there is an easier way to do it).

I would add more info, to try to make it more clear. First, I tried to identify the best distribution to fit my data, I did the following:

import scipy.stats as st
def get_best_distribution(data):
dist_names = [ 'alpha', 'anglit', 'arcsine', 'beta', 'betaprime', 'bradford',         'burr', 'cauchy', 'chi', 'chi2', 'cosine', 'dgamma', 'dweibull', 'erlang', 'expon', 'exponweib', 'exponpow', 'f', 'fatiguelife', 'fisk', 'foldcauchy', 'foldnorm', 'frechet_r', 'frechet_l', 'genlogistic', 'genpareto', 'genexpon', 'genextreme', 'gausshyper', 'gamma', 'gengamma', 'genhalflogistic', 'gilbrat', 'gompertz', 'gumbel_r', 'gumbel_l', 'halfcauchy', 'halflogistic', 'halfnorm', 'hypsecant', 'invgamma', 'invgauss', 'invweibull', 'johnsonsb', 'johnsonsu', 'ksone', 'kstwobign', 'laplace', 'logistic', 'loggamma', 'loglaplace', 'lognorm', 'lomax', 'maxwell', 'mielke', 'nakagami', 'ncx2', 'ncf', 'nct', 'norm', 'pareto', 'pearson3', 'powerlaw', 'powerlognorm', 'powernorm', 'rdist', 'reciprocal', 'rayleigh', 'rice', 'recipinvgauss', 'semicircular', 't', 'triang', 'truncexpon', 'truncnorm', 'tukeylambda', 'uniform', 'vonmises', 'wald', 'weibull_min', 'weibull_max', 'wrapcauchy']
dist_results = []
params = {}
for dist_name in dist_names:
    dist = getattr(st, dist_name)
    param = dist.fit(data)

    params[dist_name] = param
    # Applying the Kolmogorov-Smirnov test
    D, p = st.kstest(data, dist_name, args=param)
    print("p value for "+dist_name+" = "+str(p))
    dist_results.append((dist_name, p))

# select the best fitted distribution
best_dist, best_p = (max(dist_results, key=lambda item: item[1]))
# store the name of the best fit and its p value

print("Best fitting distribution: "+str(best_dist))
print("Best p value: "+ str(best_p))
print("Parameters for the best fit: "+ str(params[best_dist]))

return best_dist, best_p, params[best_dist]

That identified my distribution as a johnson unbounded.

What I have from my data is:

import scipy.stats as st

dist_name ='johnsonsu'
data= Y
dist = getattr(st, dist_name)
param = dist.fit(data)
#params[dist_name] = param
#D, p = st.kstest(data, dist_name, args=param)
print(param)
[out]: (-1.5661340035204014, 1.4899654020936477, 93994.90877721814, 55321.65122078377)

Answer 1

Enrico Fermi claimed that John von Neumann said:

With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.

On that basis, the 4-parameter unbounded Johnson distribution provides a way to transform an elephant into a standard normal distribution. The statistical question here is whether that's worth doing.* In this case, it's almost certainly not.

Many a "novice in stats" thinks that it's important to start with variables in a normal form:

I would like to transform my data (house prices) using a johnson unbounded distribution to look more gaussian.

I certainly do recall hearing, in my youth, presentations of linear regression that seemed to be based on assumptions of normality for outcomes or predictors, but that's not the case. The traditional statistical tests assume that the error term (estimated by the residuals) has a normal distribution with mean of 0, but under much weaker conditions a linear regression provides the best linear unbiased estimate (BLUE). Robust methods provide ways to assess statistical significance if the error-normality assumption is untenable.

Variable transformation can be important in regression modeling, but not typically to enforce normality of the variables themselves. Transformations of predictor variables might be important to meet the linearity assumption of the association between predictors and outcome. Restricted cubic splines provide a very flexible way to model a continuous predictor as part of a regression, more useful than anything the Johnson distributions can provide. Alternatively, modeling approaches like tree-based models will work identically regardless of a monotonic transformation of a predictor.

Transformation of an outcome variable might be needed to make residuals well enough behaved so that a BLUE can be obtained. But there's seldom a reason to start by forcing an outcome variable itself to take a normal distribution. Choosing transformations of predictors or outcomes to meet the demands of a particular problem, and knowing when to decide that something other than a linear or generalized linear regression approach is needed, are important parts of the art of modeling.

---

*The request for implementation in Python is off-topic on this site. This answer focuses on the statistical issues the request raises.