2
$\begingroup$

I have a right-skewed distribution that represents that number of "likes" on a certain car category for a number of users. What I am trying to do is a sort of classification based on z-scores. For example, if a user has made a lot of likes in favor of the given category (more than 1 standard deviation above the mean) then the user is considered engaged. Else, if the user lies 1 standard deviation below the mean then the user is not engaged at all. How can I apply this logic on my right-skewed data?

$\endgroup$
7
  • $\begingroup$ What is it about being one standard deviation above the mean that really makes it count as "engaged"? I don't imagine it's anything to do with a standard deviation per se. Is it being in the top 1/6th of the distribution? Just some arbitrary cut-off with no particular meaning? Something else? $\endgroup$ – Glen_b Sep 24 '14 at 7:26
  • $\begingroup$ OK, make it two! But the idea remains the same as asking if someone with salary 2 standard deviations above mean is a rich guy. Don't you agree? $\endgroup$ – user2295350 Sep 24 '14 at 7:32
  • $\begingroup$ What is it about being two standard deviations above the mean that makes it engaged? Or three or six or any other number? What aspect of the distribution makes you draw the line there rather than elsewhere? Understanding what you need to carry across and what you don't will suggest a way to generalize to skewed cases. $\quad$ "But the idea remains the same as asking if someone with salary 2 standard deviations above mean is a rich guy. Don't you agree?" -- I don't see why 2sds necessarily means much of anything. What about being 2sds above the mean makes it 'rich'? ... ctd $\endgroup$ – Glen_b Sep 24 '14 at 7:38
  • $\begingroup$ ctd ... what aspect of a distribution matters in terms of richness? what aspect matters for your application? $\endgroup$ – Glen_b Sep 24 '14 at 7:39
  • 1
    $\begingroup$ I am afraid everyone is focused on a wrong thing.. The key question is not who is poor or rich - the author defines that himself. The question is in the topic - CAN we use Z scores for right skewed distributions and what are the caveats of the result... $\endgroup$ – Irina10 Sep 7 '18 at 18:50
2
$\begingroup$

There are two quite distinct questions muddled together here.

Given $z =$ (value $-$ mean) / standard deviation (SD), your choices of $z > 1$ to mean "engaged" and $z < -1$ to mean "not engaged" are at best practical choices based on some context you don't give. It is difficult for anyone not familiar with your data and situation to comment, except that statistical experience might lead to warnings that your thresholds are arbitrary and are all too likely to separate people with minutely different values.

Whether the underlying distribution is symmetric or skewed is another question. If the distribution is approximately symmetric, then the fractions classified as "engaged" or "not engaged" will be about the same and can be predicted quantitatively if (and only if) you are willing to consider guesses about what the underlying distribution is (e.g. approximately symmetric binomial). If the distribution is skewed, that almost always will not hold, and it is even possible that no values lie below $z < -1$. Either way, the proof of the classification is in whether it helps some analysis somehow and you can always count how many you have in either class.

If this kind of data were mine to analyse, I would always look directly at the number of "likes" (presumably ranging from 0 to some maximum). It is not obvious to me that you need classify at all, as people are already classifying themselves; nor is it obvious that mean and SD offer the best descriptive framework.

$\endgroup$
4
  • $\begingroup$ Thanks! Users have not defined their class though. Users only 'like' things. It's me that will try to assign a class to them based on the distribution of the whole population, which is right-skewed. And yes, I hardly ever get a negative z-score. So, let me rephrase the problem once again: given a right-skewed distribution in which mean and standard deviation is known, what's the least wrong way of determining whether a user X with value V is very much or not at all interested in the particular topic (e.g. type of car)? $\endgroup$ – user2295350 Sep 23 '14 at 17:45
  • $\begingroup$ You must get some values below the mean, hence some $z < 0$! I can't see that you have a statistical question that others can answer well for you unless you define "very much or not at all interested". It is like asking how to identify "tall" or "poor" people. To be as definite as possible, I would say that the most wrong approach is to assume that distinct categories exist unless you have evidence that they do. I know my answer will appear mostly negative but I am struggling to give advice on the limited information you give. $\endgroup$ – Nick Cox Sep 23 '14 at 17:56
  • $\begingroup$ Yes, exactly, it's like asking how to identify "poor" and "rich" people! Do you thing a distribution plot would help? $\endgroup$ – user2295350 Sep 23 '14 at 18:07
  • $\begingroup$ Yes indeed; plotting your data should help. $\endgroup$ – Nick Cox Sep 23 '14 at 18:08
1
$\begingroup$

I would advise against using z-scores for markedly skewed data. For a skewed distribution, both the mean and the standard deviation are affected by the skew in a way that make the z-score results less representative of what you're trying to convey.

I think a more direct way of indicating what you what is to percentiles. Percentiles are in a sense robust against the skew in the distribution. You could use 50th percentile (the median), and say, for example, those above the median are "more engaged". Of course, you could also use the 75th percentile, or 90th percentile, as makes sense for your application.

As an example of this application, consider a classic skewed distribution, income, in this case the distribution of U.S. income from 2010. Note that the figure indicates the 50th (median), 75th, and 90th percentile of income.

As others have mentioned, the practical interpretation of these percentiles require some caution. You can confidently say that those with an income greater than the median have a higher income than the "average household"†. But practically speaking --- and living in the U.S. ---, I wouldn't say that those with greater that the median income are "high income"‡. I also wouldn't say that those with greater than 90th percentile income are "high income". It's difficult to assess, of course, because the U.S. is highly heterogeneous in this respect, so that $100,000 a year in New York City or San Francisco may not get you a place to live, whereas in some places in the country that's a really comfortable income.


† This is typical language in this application.

‡ I may have a somewhat skewed perspective since I live in New Jersey where the median income is about $80,000.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.