-1
$\begingroup$

I am working with a data set in a machine learning project, which has lots to negative values. I want to transform the distribution of my data to normal. I tried using numpy.log, but since log is only for positive values, it is producing null values. So how should I approach this issue. Below is the distribution of my variable. Also, I am confused, whether I should normalize this distribution in the first place.

enter image description here
enter image description here

$\endgroup$
  • $\begingroup$ Why do you want to transform your distribution to Normal? $\endgroup$ – jbowman Nov 20 '19 at 5:20
  • $\begingroup$ @jbowman I am thinking it can lead to improved efficiency of my model. $\endgroup$ – Saurabh Nov 20 '19 at 5:25
  • $\begingroup$ Is it not already close enough to normal for most purposes? $\endgroup$ – Michael Lew Nov 20 '19 at 5:34
  • $\begingroup$ @Micheal I think I can consider this as normal, however, I still would like to know how to deal with negative values, if I want to make the distribution normal. $\endgroup$ – Saurabh Nov 20 '19 at 5:48
  • 1
    $\begingroup$ Trying to take logarithms of negative numbers is a danger sign for your readers. You should revise this function to see why it makes no sense. You can’t usefully transform data like this, let alone “make” it normal. it is approximately symmetric to start with. But the clumping needs attention. It might reflect heterogeneity that is more of a problem for modeling than any lack of normality. There are transformations that cope with data that are both negative and positive— cube root and asinh spring to mind — but I doubt that either will help enough to be worth considering. $\endgroup$ – Nick Cox Nov 20 '19 at 8:10
1
$\begingroup$

I don't understand the need to make the distribution "Normal". You don't really give details of what you're trying to do, so I will take the simplest case I can think of (simple linear regression). You have $(y, x)$ pairs of data, and you are trying to predict $y$ from $x$.

The distribution or otherwise of $x$ is irrelevant and, depending on the assumptions you make, you may assume that the distribution of $y$ is conditionally Normal, i.e. there is a normal distribution around the line you fit, with the same variance for all fitted values of $y$.

Formally, you propose a model of the form $Y = \beta_0 + \beta_1 x + \epsilon$ and you actually assume that the errors $\epsilon$ are zero mean, constant variance Normal. You can assess whether this is a plausible assumption by looking at the residuals, $y - \hat{y}$ (where $\hat{y}$ is the predicted value of $y$ given $x$ and your parameter estimates are $\hat{\beta}_0$ and $\hat{\beta}_1$).

I know that's a lot of words, but I don't understand why you care whether your variable has a Normal distribution. I've given the default reply. In most cases you don't particularly want or care about Normal distributions. Your problem may be different, but if so, could you please add more details.

$\endgroup$
0
$\begingroup$

If you mean to z-transform your data, there is an easy implementation:

from scipy import stats
normalized_dataset = [stats.zscore(item) for item in dataset]

And yes, your AI might need standardized data! (if it doesn't standardize by itself)

$\endgroup$
  • $\begingroup$ This does not change the shape of the distribution and therefore does not appear to meet the objectives of the question. $\endgroup$ – whuber Nov 20 '19 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.