2
$\begingroup$

I have a series of dollar amounts that are highly right skewed, but are roughly log-normal. I want to put this grouped dollar amount as a predictor variable into a latent class cluster analysis. In this case I'm not sure whether transforming into log-normal makes sense for a predictor variable, as opposed to just binning the raw, skewed data.

Raw distribution:

Frequency Distribution of Raw Dollar Amounts

QQ plot of Raw Dollar Amounts

Log-normal distribution:

Frequency Distribution of Logged Dollar Amounts

QQ plot of Logged Dollar Amounts

$\endgroup$
  • 1
    $\begingroup$ Why bin at all? Clustering can use the raw values. Note that binning (or not) is a quite separate question from whether values would be better considered on a logarithmic scale. $\endgroup$ – Nick Cox Apr 4 '15 at 15:50
  • $\begingroup$ In your third sentence, did you mean to say "transforming into normal" rather than "transforming into log-normal"? $\endgroup$ – Glen_b -Reinstate Monica Jun 11 '15 at 23:35
1
$\begingroup$

Ultimately, I discovered that there is no "correct' answer for this. One issue I had w/ trying to create lognormal data was that I had numerous data points with 0. Obviously negative infinity doesn't bin very well, so i had to force those values into the first bin. I ended up using Jenks natural breaks on the raw data through R's prabclus package.

I was using this break method to create a categorical variable for latent class analysis. I think that if I were predicting on the dollar amount, some kind of transformation to log-normal would make sense. But scaling for the purpose of scaling probably doesn't make sense in this case. Additionally, after running iterations of LCA with & without the Amount as a classifier, I saw that there was little change in the shape or number of clusters, so my off-the-cuff determination was, it wasn't particularly meaningful. I guess a more robust way to determine that would be Principal Components.

$\endgroup$
1
$\begingroup$

First off, there are two broad types of latent class models: supervised and unsupervised. The unsupervised stream is the earliest version and dates back to the post-WWII world and the work of Paul Lazarsfeld, a Columbia sociologist. Lazersfeld's tradition was later picked up by more recent sociologists like Leo Goodman and the late Clifford Clogg. Clogg formalized his views with an LC software tool called MLLSA (pronounced like 'melissa'). In unsupervised LC, the inputs are categorical and the "replications" are based on the cross-classification of the levels taken combinatorily across all factors. Supervised LC, on the other hand, typically takes the form of a finite mixture model and relies on maximum likelihood estimation, which is invariant to scaling. This workstream began with Heckman, saw implementation in sociology with "grade-of-membership" models and in marketing with late 80s papers by Bill Dillon (LADI, latent discriminant models) and Wagner Kamakura. "Replications" are typically at the level of the primary unit of analysis -- in marketing this would refer to consumers -- and over time. The focus is on partitioning the heterogeneity in the residual output from the model.

To me, it sounds like the OP's model is unsupervised, requiring purely categorical inputs and that is motivating the bucketing of revenues. If this is so, then my recommendation is to consider a couple of options. First, add +1 constant to all revenue values to shift the PDF from zero to one and then take the natural log. Next, at the same time you've indicated that the natural log transform of revenues didn't sufficiently compress the "tail" of the revenue PDF. If so, then there are other, "stronger" functions that can be applied. For instance, try the inverse hyperbolic sine or Lambert's W both of which can transform even hugely heavy-tailed PDFs into symmetry. Then, if you must, bin that new variable.

Otherwise, you can consider finding tools that don't limit you to categorical-only data inputs. In other words, software exists that will implement a true finite mixture model with mixed scale types even for unsupervised LC models. Not being an R guru, my preference for all things LC is to use Latent Gold. This isn't a sales pitch, just an honest assessment that the functionality of LG software is as close to state of the art in LC models as canned software can get. And in this instance, my best information is that nothing in R even comes close to LG in range, power and flexibility.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.