3
$\begingroup$

I am doing bioinformatics and I am trying to fit some values to a log-normal distribution with python's sciPy version 0.11. According to the skew of the resulting distribution, I would like to make a 0-1 decision, i.e. if skew is positive, give a "0" value, as most of the data is on the left, while, if skew is negative, give a "1" value, as most of the data is on the right.

While trying to implement that, I am confused as to which formula to use for the skew. From Wikipedia, the skew is equal to: $(e^{\sigma^2}+2)\sqrt{e^{\sigma^2}-1}$, but this can never be a negative number. Using sciPy's moment method like this: scipy.stats.moment(data,3)/(std**3), seems to yield negative numbers, but then I found out this is the non-central moment, so I am afraid it is wrong.

Secondly, I have looked into this very useful question here, on how to actually fit data to the log-normal distribution. I have tried to make a toy example, where I generate 50 numbers on the interval $0.1-9$ and 1000 numbers on the interval $9-12$. So, I would expect that this distribution should have a negative skew. The code is as follows:

import numpy as np
import scipy.stats as s
from matplotlib import pyplot as plt
test1 = np.linspace(0.1,9,50)
test2 = np.linspace(9,12,1000)
test = list(test1) + list(test2)
shape,loc,scale = s.lognorm.fit(test,floc=0)
distro = s.lognorm(shape,loc,scale)
x = np.linspace(0.1,20,200)
plt.plot(x,distro.pdf(x))
plt.plot(x,distro.cdf(x))
plt.axvline(x=scale,color='r',ls='--')
plt.legend(('PDF','CDF',r'$\mu$'), loc='upper left')
plt.hist(test,bins=100,normed=True,alpha=0.4)

And the corresponding image: A toy example histogram and fitted log-normal distribution

My question is, does it really take that much to shift a log-normal distribution to the right? What I have done here seems to spread it wide, but shift it only a little. Should I use a different measure for my purposes?

$\endgroup$
3
$\begingroup$

According to the skew of the resulting distribution, I would like to make a 0-1 decision, i.e. if skew is positive, give a "0" value, as most of the data is on the left, while, if skew is negative, give a "1" value, as most of the data is on the right.

As a general statement, this is not true. It's often the case, but it's trivial to find counterexamples (and unfortunately many elementary books insist on making statements almost exactly like yours, to eventual substantial confusion when a real-world counterexample appears).

In your question, you're using third-moment skewness as your skewness measure, but statements that 'most of the data is on the left'/'most of the data is on the right' relates to a different kind of skewness, the second Pearson skewness coefficient.

The two measures can disagree about the sign of skewness in a population.

Consider, for example, a Poisson with mean 0.7. It has third moment skewness of about 1.195, but it has second Pearson skewness (/mean-median skewness) of $3(0.7-1)/\sqrt 0.7$, or about -1.076 (that is, in spite of having positive third moment skewness, more of its probability is above the mean than below it). [That's a simple example I only came up with a couple of days ago in response to another question; it only took a few minutes to come up with several such examples, but that one's my favourite of the counterexamples, not least because "Poisson(0.7)" is so simple to picture, so easy to remember and unambiguously convey.]

From Wikipedia, the skew is equal to: $(e^{σ^2}+2)\sqrt{e^{σ^2}−1}$, but this can never be a negative number. Using sciPy's moment method like this: scipy.stats.moment(data,3)/(std**3)

You're confusing two different things! The first is a population quantity and the second is a sample quantity.

You can have one positive while the other is negative, just by chance.

Here's an example. This is a sample I just generated from a lognormal distribution ($n=30$, with parameters $\mu=0$ and $\sigma=0.1$), which happens to have negative sample skewness:

 0.139  0.086  0.046 -0.084  0.020  0.050 -0.041  0.080 -0.048  0.076 -0.050  0.023  
 0.063 -0.210  0.011 -0.343 -0.016 -0.005  0.123  0.044 -0.026  0.048  0.107  0.066  
 0.089 -0.047  0.175 -0.092 -0.095  0.020

In fact with those parameters and that sample size, the sample third moment is negative almost half the time.

$\endgroup$
  • $\begingroup$ Thank you very much for the quick reply! I knew I was confusing something. This really clarified it! I don't have enough rep to vote your answer up, but will do so asap1 $\endgroup$ – Dima1982 Nov 11 '13 at 22:14
  • $\begingroup$ Relating to your edit, I was mixing up the population-sample quantities. I am trying to discretize gene expression data (that usually fit a log-normal distribution). So, in a sample of expressions from multiple cells, if a gene is "mostly on", give a value of 1, 0 otherwise. That was my initial idea, at least. $\endgroup$ – Dima1982 Nov 11 '13 at 22:20
  • $\begingroup$ How does discretizing a variable relate to your questions about skewness? I am afraid I still don't understand enough about what you're doing to answer the final question about shifting to the right (on the other hand it sounds like you're satisfied so maybe it doesn't matter now). I seem to be done editing for the moment. $\endgroup$ – Glen_b -Reinstate Monica Nov 11 '13 at 22:29
  • $\begingroup$ Actually, I am trying to make a decision like this: for one gene, if distribution is shifted to the right, assume 1 (gene is mostly 'on') and for a new value evaluate cdf(z-score-of-value). If cdf <0.2 then 1, else 0. If skewed to the left (gene mostly 'off'), then assume 0. For a new value evaluate 1-cdf(z-score-of-value) and if 1-cdf < 0.2, then assign 0, else 1. The extra example, really clarifies it. I think that if I use the Pearson moment, it should be OK for my purposes. I needed a measure that relates "most of the probability" to the mean. $\endgroup$ – Dima1982 Nov 11 '13 at 22:36
  • $\begingroup$ Okay. I'll remove the last part of my answer. (Still not completely clear on what's going on with that part of your question though.) $\endgroup$ – Glen_b -Reinstate Monica Nov 11 '13 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.